public inbox for [email protected]help / color / mirror / Atom feed
[PATCH v21 1/7] Built-in compression method 119+ messages / 3 participants [nested] [flat]
* [PATCH v21 1/7] Built-in compression method @ 2020-09-08 09:54 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2020-09-08 09:54 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 99 +++++++ configure.ac | 22 ++ contrib/test_decoding/expected/ddl.out | 50 ++-- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 ++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 72 +++-- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++-- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 147 ++++++++++ .../access/compression/compress_pglz.c | 131 +++++++++ .../access/compression/compressamapi.c | 61 +++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 6 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 26 +- src/backend/commands/createas.c | 7 + src/backend/commands/matview.c | 7 + src/backend/commands/tablecmds.c | 101 ++++++- src/backend/executor/nodeModifyTable.c | 83 ++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 8 + .../replication/logical/reorderbuffer.c | 1 + src/backend/utils/adt/pg_upgrade_support.c | 10 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 46 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 46 +++- src/bin/pg_dump/pg_dump.h | 2 + src/bin/psql/describe.c | 42 ++- src/include/access/compressamapi.h | 65 +++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 22 +- src/include/catalog/binary_upgrade.h | 2 + src/include/catalog/pg_am.dat | 7 +- src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 10 +- src/include/catalog/pg_proc.dat | 24 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 3 +- src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/alter_table.out | 10 +- src/test/regress/expected/compression.out | 176 ++++++++++++ src/test/regress/expected/compression_1.out | 171 ++++++++++++ src/test/regress/expected/copy2.out | 8 +- src/test/regress/expected/create_table.out | 142 +++++----- .../regress/expected/create_table_like.out | 88 +++--- src/test/regress/expected/domain.out | 16 +- src/test/regress/expected/foreign_data.out | 258 +++++++++--------- src/test/regress/expected/identity.out | 16 +- src/test/regress/expected/inherit.out | 138 +++++----- src/test/regress/expected/insert.out | 118 ++++---- src/test/regress/expected/matview.out | 80 +++--- src/test/regress/expected/psql.out | 108 ++++---- src/test/regress/expected/publication.out | 40 +-- .../regress/expected/replica_identity.out | 14 +- src/test/regress/expected/rowsecurity.out | 16 +- src/test/regress/expected/rules.out | 30 +- src/test/regress/expected/stats_ext.out | 10 +- src/test/regress/expected/update.out | 16 +- src/test/regress/output/tablespace.source | 16 +- src/test/regress/parallel_schedule | 3 + src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 74 +++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 81 files changed, 2193 insertions(+), 668 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/backend/access/compression/compressamapi.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index e202697bbf..a7b4317b90 100755 --- a/configure +++ b/configure @@ -698,6 +698,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -865,6 +866,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld enable_largefile ' @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --without-lz4 do not use lz4 --with-gnu-ld assume the C compiler uses GNU ld [default=no] Some influential environment variables: @@ -8596,6 +8599,35 @@ fi +# +# lz4 +# + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + : + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12087,6 +12119,59 @@ fi fi +if test "$with_lz4" = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "lz4 library not found +If you have lz4 already installed, see config.log for details on the +failure. It is possible the compiler isn't looking in the proper directory. +Use --without-lz4 to disable lz4 support." "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13290,6 +13375,20 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + +else + as_fn_error $? "lz4 header not found +If you have lz4 already installed, see config.log for details on the +failure. It is possible the compiler isn't looking in the proper directory. +Use --without-lz4 to disable lz4 support." "$LINENO" 5 +fi + + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index a5ad072ee4..ce29d6e97f 100644 --- a/configure.ac +++ b/configure.ac @@ -995,6 +995,13 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# lz4 +# +PGAC_ARG_BOOL(with, lz4, yes, + [do not use lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1182,6 +1189,14 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_LIB(lz4, LZ4_compress, [], + [AC_MSG_ERROR([lz4 library not found +If you have lz4 already installed, see config.log for details on the +failure. It is possible the compiler isn't looking in the proper directory. +Use --without-lz4 to disable lz4 support.])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1397,6 +1412,13 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADER(lz4.h, [], [AC_MSG_ERROR([lz4 header not found +If you have lz4 already installed, see config.log for details on the +failure. It is possible the compiler isn't looking in the proper directory. +Use --without-lz4 to disable lz4 support.])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..5bf614cc8a 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from its parent table, however a different compression method can be set + for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..d514c7f8c1 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This clause adds the compression method to a column. The Compression + method can be set from available compression methods. The built-in + methods are <literal>pglz</literal> and <literal>lz4</literal>. + If no compression method is specified, then compressible types will have + the default compression method <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..c90464053c 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,47 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_handler - get the compression handler routines * - * Decompress a compressed version of a varlena datum + * helper function for toast_decompress_datum and toast_decompress_datum_slice */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) { - struct varlena *result; + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +512,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); - - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..c4814ef911 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o compressamapi.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..e8b035d138 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,147 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2016-2020, PostgreSQL Global Development Group + * Portions Copyright (c) 1990-1993, Regents of the University of California + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "postgres.h" +#include "access/compressamapi.h" +#include "access/toast_internals.h" +#include "fmgr.h" +#include "utils/builtins.h" + +#ifdef HAVE_LIBLZ4 +#include "lz4.h" +#endif + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + max_size = LZ4_compressBound(VARSIZE_ANY_EXHDR(value)); + tmp = (struct varlena *) palloc(max_size + TOAST_COMPRESS_HDRSZ); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + TOAST_COMPRESS_HDRSZ, + valsize, max_size); + if (len <= 0) + { + pfree(tmp); + elog(ERROR, "lz4: could not compress data"); + } + + SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + result = (struct varlena *) palloc(TOAST_COMPRESS_RAWSIZE(value) + VARHDRSZ); + SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(value) + VARHDRSZ); + + rawsize = LZ4_decompress_safe(TOAST_COMPRESS_RAWDATA(value), + VARDATA(result), + VARSIZE(value) - TOAST_COMPRESS_HDRSZ, + TOAST_COMPRESS_RAWSIZE(value)); + if (rawsize < 0) + elog(ERROR, "lz4: compressed data is corrupted"); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + result = (struct varlena *) palloc(TOAST_COMPRESS_RAWSIZE(value) + VARHDRSZ); + + rawsize = LZ4_decompress_safe_partial(TOAST_COMPRESS_RAWDATA(value), + VARDATA(result), + VARSIZE(value) - TOAST_COMPRESS_HDRSZ, + slicelength, + TOAST_COMPRESS_RAWSIZE(value)); + if (rawsize < 0) + elog(ERROR, "lz4: compressed data is corrupted"); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..578be6f1dd --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,131 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Copyright (c) 2015-2018, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "access/toast_internals.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + TOAST_COMPRESS_HDRSZ); + + len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), + valsize, + TOAST_COMPRESS_RAWDATA(tmp), + NULL); + + if (len >= 0) + { + SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); + return tmp; + } + + pfree(tmp); + + return NULL; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(TOAST_COMPRESS_RAWSIZE(value) + VARHDRSZ); + SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(value) + VARHDRSZ); + + rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(value), + TOAST_COMPRESS_SIZE(value), + VARDATA(result), + TOAST_COMPRESS_RAWSIZE(value), true); + + if (rawsize < 0) + elog(ERROR, "pglz: compressed data is corrupted"); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(value), + VARSIZE(value) - TOAST_COMPRESS_HDRSZ, + VARDATA(result), + slicelength, false); + + if (rawsize < 0) + elog(ERROR, "pglz: compressed data is corrupted"); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/compression/compressamapi.c b/src/backend/access/compression/compressamapi.c new file mode 100644 index 0000000000..663102c8d2 --- /dev/null +++ b/src/backend/access/compression/compressamapi.c @@ -0,0 +1,61 @@ +/*------------------------------------------------------------------------- + * + * compression/compressamapi.c + * Functions for compression methods + * + * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * + * IDENTIFICATION + * src/backend/access/compression/compressamapi.c + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "access/compressamapi.h" +#include "access/htup.h" +#include "access/htup_details.h" +#include "access/reloptions.h" +#include "access/table.h" +#include "utils/fmgroids.h" +#include "utils/syscache.h" + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "Invalid compression method id %d", cmid); + } +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index 009e215da1..44fa2ca7b4 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -181,6 +181,9 @@ my $C_COLLATION_OID = my $PG_CATALOG_NAMESPACE = Catalog::FindDefinedSymbolFromData($catalog_data{pg_namespace}, 'PG_CATALOG_NAMESPACE'); +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. @@ -808,6 +811,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b8cd35e995..e6040923dc 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -347,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..668e43faed 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -34,6 +34,8 @@ static Oid lookup_am_handler_func(List *handler_name, char amtype); static const char *get_am_type_string(char amtype); +/* Set by pg_upgrade_support functions */ +Oid binary_upgrade_next_pg_am_oid = InvalidOid; /* * CreateAccessMethod @@ -83,7 +85,19 @@ CreateAccessMethod(CreateAmStmt *stmt) memset(values, 0, sizeof(values)); memset(nulls, false, sizeof(nulls)); - amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid); + if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_pg_am_oid)) + { + /* acoid should be found in some cases */ + if (binary_upgrade_next_pg_am_oid < FirstNormalObjectId && + (!OidIsValid(amoid) || binary_upgrade_next_pg_am_oid != amoid)) + elog(ERROR, "could not link to built-in attribute compression"); + + amoid = binary_upgrade_next_pg_am_oid; + binary_upgrade_next_pg_am_oid = InvalidOid; + } + else + amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid); + values[Anum_pg_am_oid - 1] = ObjectIdGetDatum(amoid); values[Anum_pg_am_amname - 1] = DirectFunctionCall1(namein, CStringGetDatum(stmt->amname)); @@ -175,6 +189,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..59690ce854 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -581,6 +581,13 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + CompareCompressionMethodAndDecompress(slot, myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..41c66c9e61 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -486,6 +486,13 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + CompareCompressionMethodAndDecompress(slot, myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..d7f4489c57 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,32 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * Get compression method Oid for the attribute. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression != NULL) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + return InvalidOid; + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + return get_compression_am_oid(compression, false); +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 5d90337498..684e836071 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,12 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_internals.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2039,78 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. + */ +void +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool isnull = false; + bool decompressed_any = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return; + + /* + * Loop for all the attributes in the tuple and check if any of the + * attribute is compressed in the source tuple and its compression method + * is not same as the target compression method then we need to decompress + * it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value = (struct varlena *) + DatumGetPointer(slot_getattr(slot, attnum, &isnull)); + + /* nothing to be done, if the value is null */ + if (isnull) + continue; + + /* nothing to be done. if it is not compressed */ + if (!VARATT_IS_COMPRESSED(new_value)) + continue; + + /* + * Get the compression method stored in the toast header and + * compare with the compression method of the target. + */ + if (targetTupDesc->attrs[i].attcompression != + CompressionIdToOid(TOAST_COMPRESS_METHOD(new_value))) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the field in the source tuple then free + * the existing tuple. + */ + if (decompressed_any) + { + /* deform complete tuple before we clear the existing tuple */ + slot_getallattrs(slot); + + ExecClearTuple(slot); + slot->tts_nvalid = natts; + slot->tts_flags &= ~TTS_FLAG_EMPTY; + } +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2319,14 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + CompareCompressionMethodAndDecompress( + slot, resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index ba3ccc712c..3ea32323cd 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2932,6 +2932,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index a2ef853dc2..227bb07bbd 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2598,6 +2598,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 6be19916fc..c6ba9e1b05 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3858,6 +3858,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 8392be6d44..187d3570e5 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2865,6 +2865,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 7574d545e0..355e273192 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -595,6 +595,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -630,9 +632,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3420,11 +3422,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3433,8 +3436,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3479,6 +3482,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3709,6 +3720,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15237,6 +15249,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15755,6 +15768,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..e2ac159aa2 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if (table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index 5a62ab8bbc..43eee8a4b7 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++ b/src/backend/replication/logical/reorderbuffer.c @@ -87,6 +87,7 @@ #include "access/detoast.h" #include "access/heapam.h" #include "access/rewriteheap.h" +#include "access/toast_internals.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog_internal.h" diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c index a575c95079..f026104c01 100644 --- a/src/backend/utils/adt/pg_upgrade_support.c +++ b/src/backend/utils/adt/pg_upgrade_support.c @@ -128,6 +128,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS) PG_RETURN_VOID(); } +Datum +binary_upgrade_set_next_pg_am_oid(PG_FUNCTION_ARGS) +{ + Oid amoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_pg_am_oid = amoid; + PG_RETURN_VOID(); +} + Datum binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS) { diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..aff5b7e95e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_internals.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,50 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + char *compression; + int typlen; + struct varlena *varvalue; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + /* + * If it is not a varlena type or the attribute is not compressed then + * return NULL. + */ + if ((typlen != -1) || !VARATT_IS_COMPRESSED(value)) + PG_RETURN_NULL(); + + varvalue = (struct varlena *) DatumGetPointer(value); + + compression = + get_am_name(CompressionIdToOid(TOAST_COMPRESS_METHOD(varvalue))); + + PG_RETURN_TEXT_P(cstring_to_text(compression)); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 9d0056a569..5e168a3c04 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 39da742e32..becf565cdd 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -13016,6 +13035,11 @@ dumpAccessMethod(Archive *fout, AccessMethodInfo *aminfo) qamname = pg_strdup(fmtId(aminfo->dobj.name)); + if (dopt->binary_upgrade && aminfo->amtype == AMTYPE_COMPRESSION) + appendPQExpBuffer(q, + "SELECT pg_catalog.binary_upgrade_set_next_pg_am_oid('%u'::pg_catalog.oid);\n", + aminfo->dobj.catId.oid); + appendPQExpBuffer(q, "CREATE ACCESS METHOD %s ", qamname); switch (aminfo->amtype) @@ -15805,6 +15829,7 @@ dumpTableSchema(Archive *fout, TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15829,6 +15854,9 @@ dumpTableSchema(Archive *fout, TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15864,6 +15892,22 @@ dumpTableSchema(Archive *fout, TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Compression + * + * In binary-upgrade mode, compression is assigned by + * ALTER. Even if we're skipping compression the attribute + * will get default compression. It's the task for ALTER + * command to restore compression info. + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + has_non_default_compression) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 1290f9659b..53ece7abd7 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -327,6 +327,8 @@ typedef struct _tableInfo bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ + /* * Stuff computed only for dumpable tables. */ diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..4f82929fdb 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,27 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + bool mustfree = false; + const int trunclen = 100; + char *val = PQgetvalue(res, i, attcompression_col); + + /* truncate the options if they're too long */ + if (strlen(val) > trunclen + 3) + { + char *trunc = pg_malloc0(trunclen + 4); + strncpy(trunc, val, trunclen); + strncpy(trunc + trunclen, "...", 4); + + val = trunc; + mustfree = true; + } + + printTableAddCell(&cont, val, false, mustfree); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..8226ae0596 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,65 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Copyright (c) 2015-2017, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* access/compression/compressamapi.c */ +extern CompressionId CompressionOidToId(Oid cmoid); +extern Oid CompressionIdToOid(CompressionId cmid); + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..84ba303e7e 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,33 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; +#define RAWSIZEMASK (0x3FFFFFFFU) + /* * Utilities for manipulation of header information for compressed * toast entries. + * + * Since version 14 TOAST_COMPRESS_SET_RAWSIZE also marks compressed + * varlenas as custom compressed. Such varlenas will contain 0x02 (0b10) in + * two highest bits. */ #define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) +#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->info & RAWSIZEMASK) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> 30) #define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) #define TOAST_COMPRESS_RAWDATA(ptr) \ (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= RAWSIZEMASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << 30); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h index f6e82e7ac5..9d65bae238 100644 --- a/src/include/catalog/binary_upgrade.h +++ b/src/include/catalog/binary_upgrade.h @@ -26,6 +26,8 @@ extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid; extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid; extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid; +extern PGDLLIMPORT Oid binary_upgrade_next_pg_am_oid; + extern PGDLLIMPORT bool binary_upgrade_record_init_privs; #endif /* BINARY_UPGRADE_H */ diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..00362d9db3 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,10 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, - +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index 358f5aac8f..4a5c249ee9 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX(pg_am_oid_index, 2652, on pg_am using btree(oid oid_ops)); */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 059dec3647..a6cb1b4e5b 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -156,6 +156,14 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation */ Oid attcollation; + /* + * Oid of the compression method that will be used for compressing the value + * for this attribute. For compressible types this must always be a + * valid Oid irrespective of what is the current value of the attstorage. + * And for the incompressible types this must be InvalidOid. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -183,7 +191,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index b5f52d4e4a..7458ed7d0d 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7087,6 +7096,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7257,6 +7270,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, @@ -10749,6 +10769,10 @@ proname => 'binary_upgrade_set_next_pg_authid_oid', provolatile => 'v', proparallel => 'r', prorettype => 'void', proargtypes => 'oid', prosrc => 'binary_upgrade_set_next_pg_authid_oid' }, +{ oid => '1137', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_pg_am_oid', provolatile => 'v', + proparallel => 'r', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_pg_am_oid' }, { oid => '3591', descr => 'for use by pg_upgrade', proname => 'binary_upgrade_create_empty_extension', proisstrict => 'f', provolatile => 'v', proparallel => 'u', prorettype => 'void', diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 56da2913bd..984002ba21 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -625,6 +625,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 758c3ca097..36d32f94aa 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -616,5 +616,6 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern void CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index caed683ba9..ded2ad9692 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -510,6 +510,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index dc2bb40926..232c9f1248 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 8c554e1f69..89bd2fd7a8 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -87,6 +87,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index f4d9f3b408..47f114b8a0 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..90c7454d2a 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -55,9 +55,9 @@ /* * struct varatt_external is a traditional "TOAST pointer", that is, the * information needed to fetch a Datum stored out-of-line in a TOAST table. - * The data is compressed if and only if va_extsize < va_rawsize - VARHDRSZ. - * This struct must not contain any padding, because we sometimes compare - * these pointers using memcmp. + * The data is compressed if and only if size in + * va_extinfo < va_rawsize - VARHDRSZ. This struct must not contain any + * padding, because we sometimes compare these pointers using memcmp. * * Note that this information is stored unaligned within actual tuples, so * you need to memcpy from the tuple into a local struct variable before @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -280,8 +281,11 @@ typedef struct #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & 0x3FFFFFFF) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> 30) /* Externally visible macros */ diff --git a/src/test/regress/expected/alter_table.out b/src/test/regress/expected/alter_table.out index 0ce6ee4622..138df4be93 100644 --- a/src/test/regress/expected/alter_table.out +++ b/src/test/regress/expected/alter_table.out @@ -2196,11 +2196,11 @@ where oid = 'test_storage'::regclass; create index test_storage_idx on test_storage (b, a); alter table test_storage alter column a set storage external; \d+ test_storage - Table "public.test_storage" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | external | | - b | integer | | | 0 | plain | | + Table "public.test_storage" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | external | pglz | | + b | integer | | | 0 | plain | | | Indexes: "test_storage_idx" btree (b, a) diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..4dea269fc3 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,176 @@ +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..a50c7679bb --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,171 @@ +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: not built with lz4 support +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- +(0 rows) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +-------- +(0 rows) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- +(0 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- +(0 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: not built with lz4 support +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- +(0 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out index c64f0719e7..1778c15c7a 100644 --- a/src/test/regress/expected/copy2.out +++ b/src/test/regress/expected/copy2.out @@ -511,10 +511,10 @@ begin end $$ language plpgsql immutable; alter table check_con_tbl add check (check_con_function(check_con_tbl.*)); \d+ check_con_tbl - Table "public.check_con_tbl" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - f1 | integer | | | | plain | | + Table "public.check_con_tbl" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | Check constraints: "check_con_tbl_check" CHECK (check_con_function(check_con_tbl.*)) diff --git a/src/test/regress/expected/create_table.out b/src/test/regress/expected/create_table.out index ed8c01b8de..3105115fee 100644 --- a/src/test/regress/expected/create_table.out +++ b/src/test/regress/expected/create_table.out @@ -498,11 +498,11 @@ Partition key: RANGE (a oid_ops, plusone(b), c, d COLLATE "C") Number of partitions: 0 \d+ partitioned2 - Partitioned table "public.partitioned2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | integer | | | | plain | | - b | text | | | | extended | | + Partitioned table "public.partitioned2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | integer | | | | plain | | | + b | text | | | | extended | pglz | | Partition key: RANGE (((a + 1)), substr(b, 1, 5)) Number of partitions: 0 @@ -511,11 +511,11 @@ ERROR: no partition of relation "partitioned2" found for row DETAIL: Partition key of the failing row contains ((a + 1), substr(b, 1, 5)) = (2, hello). CREATE TABLE part2_1 PARTITION OF partitioned2 FOR VALUES FROM (-1, 'aaaaa') TO (100, 'ccccc'); \d+ part2_1 - Table "public.part2_1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | integer | | | | plain | | - b | text | | | | extended | | + Table "public.part2_1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | integer | | | | plain | | | + b | text | | | | extended | pglz | | Partition of: partitioned2 FOR VALUES FROM ('-1', 'aaaaa') TO (100, 'ccccc') Partition constraint: (((a + 1) IS NOT NULL) AND (substr(b, 1, 5) IS NOT NULL) AND (((a + 1) > '-1'::integer) OR (((a + 1) = '-1'::integer) AND (substr(b, 1, 5) >= 'aaaaa'::text))) AND (((a + 1) < 100) OR (((a + 1) = 100) AND (substr(b, 1, 5) < 'ccccc'::text)))) @@ -552,11 +552,11 @@ select * from partitioned where partitioned = '(1,2)'::partitioned; (2 rows) \d+ partitioned1 - Table "public.partitioned1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | - b | integer | | | | plain | | + Table "public.partitioned1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | + b | integer | | | | plain | | | Partition of: partitioned FOR VALUES IN ('(1,2)') Partition constraint: (((partitioned1.*)::partitioned IS DISTINCT FROM NULL) AND ((partitioned1.*)::partitioned = '(1,2)'::partitioned)) @@ -609,10 +609,10 @@ CREATE TABLE part_p2 PARTITION OF list_parted FOR VALUES IN (2); CREATE TABLE part_p3 PARTITION OF list_parted FOR VALUES IN ((2+1)); CREATE TABLE part_null PARTITION OF list_parted FOR VALUES IN (null); \d+ list_parted - Partitioned table "public.list_parted" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | + Partitioned table "public.list_parted" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | Partition key: LIST (a) Partitions: part_null FOR VALUES IN (NULL), part_p1 FOR VALUES IN (1), @@ -1049,21 +1049,21 @@ create table test_part_coll_cast2 partition of test_part_coll_posix for values f drop table test_part_coll_posix; -- Partition bound in describe output \d+ part_b - Table "public.part_b" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | not null | 1 | plain | | + Table "public.part_b" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | not null | 1 | plain | | | Partition of: parted FOR VALUES IN ('b') Partition constraint: ((a IS NOT NULL) AND (a = 'b'::text)) -- Both partition bound and partition key in describe output \d+ part_c - Partitioned table "public.part_c" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | not null | 0 | plain | | + Partitioned table "public.part_c" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | not null | 0 | plain | | | Partition of: parted FOR VALUES IN ('c') Partition constraint: ((a IS NOT NULL) AND (a = 'c'::text)) Partition key: RANGE (b) @@ -1071,11 +1071,11 @@ Partitions: part_c_1_10 FOR VALUES FROM (1) TO (10) -- a level-2 partition's constraint will include the parent's expressions \d+ part_c_1_10 - Table "public.part_c_1_10" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | not null | 0 | plain | | + Table "public.part_c_1_10" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | not null | 0 | plain | | | Partition of: part_c FOR VALUES FROM (1) TO (10) Partition constraint: ((a IS NOT NULL) AND (a = 'c'::text) AND (b IS NOT NULL) AND (b >= 1) AND (b < 10)) @@ -1104,46 +1104,46 @@ Number of partitions: 3 (Use \d+ to list them.) CREATE TABLE range_parted4 (a int, b int, c int) PARTITION BY RANGE (abs(a), abs(b), c); CREATE TABLE unbounded_range_part PARTITION OF range_parted4 FOR VALUES FROM (MINVALUE, MINVALUE, MINVALUE) TO (MAXVALUE, MAXVALUE, MAXVALUE); \d+ unbounded_range_part - Table "public.unbounded_range_part" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | - b | integer | | | | plain | | - c | integer | | | | plain | | + Table "public.unbounded_range_part" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | + b | integer | | | | plain | | | + c | integer | | | | plain | | | Partition of: range_parted4 FOR VALUES FROM (MINVALUE, MINVALUE, MINVALUE) TO (MAXVALUE, MAXVALUE, MAXVALUE) Partition constraint: ((abs(a) IS NOT NULL) AND (abs(b) IS NOT NULL) AND (c IS NOT NULL)) DROP TABLE unbounded_range_part; CREATE TABLE range_parted4_1 PARTITION OF range_parted4 FOR VALUES FROM (MINVALUE, MINVALUE, MINVALUE) TO (1, MAXVALUE, MAXVALUE); \d+ range_parted4_1 - Table "public.range_parted4_1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | - b | integer | | | | plain | | - c | integer | | | | plain | | + Table "public.range_parted4_1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | + b | integer | | | | plain | | | + c | integer | | | | plain | | | Partition of: range_parted4 FOR VALUES FROM (MINVALUE, MINVALUE, MINVALUE) TO (1, MAXVALUE, MAXVALUE) Partition constraint: ((abs(a) IS NOT NULL) AND (abs(b) IS NOT NULL) AND (c IS NOT NULL) AND (abs(a) <= 1)) CREATE TABLE range_parted4_2 PARTITION OF range_parted4 FOR VALUES FROM (3, 4, 5) TO (6, 7, MAXVALUE); \d+ range_parted4_2 - Table "public.range_parted4_2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | - b | integer | | | | plain | | - c | integer | | | | plain | | + Table "public.range_parted4_2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | + b | integer | | | | plain | | | + c | integer | | | | plain | | | Partition of: range_parted4 FOR VALUES FROM (3, 4, 5) TO (6, 7, MAXVALUE) Partition constraint: ((abs(a) IS NOT NULL) AND (abs(b) IS NOT NULL) AND (c IS NOT NULL) AND ((abs(a) > 3) OR ((abs(a) = 3) AND (abs(b) > 4)) OR ((abs(a) = 3) AND (abs(b) = 4) AND (c >= 5))) AND ((abs(a) < 6) OR ((abs(a) = 6) AND (abs(b) <= 7)))) CREATE TABLE range_parted4_3 PARTITION OF range_parted4 FOR VALUES FROM (6, 8, MINVALUE) TO (9, MAXVALUE, MAXVALUE); \d+ range_parted4_3 - Table "public.range_parted4_3" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | - b | integer | | | | plain | | - c | integer | | | | plain | | + Table "public.range_parted4_3" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | + b | integer | | | | plain | | | + c | integer | | | | plain | | | Partition of: range_parted4 FOR VALUES FROM (6, 8, MINVALUE) TO (9, MAXVALUE, MAXVALUE) Partition constraint: ((abs(a) IS NOT NULL) AND (abs(b) IS NOT NULL) AND (c IS NOT NULL) AND ((abs(a) > 6) OR ((abs(a) = 6) AND (abs(b) >= 8))) AND (abs(a) <= 9)) @@ -1175,11 +1175,11 @@ SELECT obj_description('parted_col_comment'::regclass); (1 row) \d+ parted_col_comment - Partitioned table "public.parted_col_comment" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+--------------- - a | integer | | | | plain | | Partition key - b | text | | | | extended | | + Partitioned table "public.parted_col_comment" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+--------------- + a | integer | | | | plain | | | Partition key + b | text | | | | extended | pglz | | Partition key: LIST (a) Number of partitions: 0 @@ -1188,10 +1188,10 @@ DROP TABLE parted_col_comment; CREATE TABLE arrlp (a int[]) PARTITION BY LIST (a); CREATE TABLE arrlp12 PARTITION OF arrlp FOR VALUES IN ('{1}', '{2}'); \d+ arrlp12 - Table "public.arrlp12" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+-----------+-----------+----------+---------+----------+--------------+------------- - a | integer[] | | | | extended | | + Table "public.arrlp12" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-----------+-----------+----------+---------+----------+-------------+--------------+------------- + a | integer[] | | | | extended | pglz | | Partition of: arrlp FOR VALUES IN ('{1}', '{2}') Partition constraint: ((a IS NOT NULL) AND ((a = '{1}'::integer[]) OR (a = '{2}'::integer[]))) @@ -1201,10 +1201,10 @@ create table boolspart (a bool) partition by list (a); create table boolspart_t partition of boolspart for values in (true); create table boolspart_f partition of boolspart for values in (false); \d+ boolspart - Partitioned table "public.boolspart" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | boolean | | | | plain | | + Partitioned table "public.boolspart" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | boolean | | | | plain | | | Partition key: LIST (a) Partitions: boolspart_f FOR VALUES IN (false), boolspart_t FOR VALUES IN (true) diff --git a/src/test/regress/expected/create_table_like.out b/src/test/regress/expected/create_table_like.out index 10d17be23c..4da878ee1c 100644 --- a/src/test/regress/expected/create_table_like.out +++ b/src/test/regress/expected/create_table_like.out @@ -325,32 +325,32 @@ CREATE TABLE ctlt4 (a text, c text); ALTER TABLE ctlt4 ALTER COLUMN c SET STORAGE EXTERNAL; CREATE TABLE ctlt12_storage (LIKE ctlt1 INCLUDING STORAGE, LIKE ctlt2 INCLUDING STORAGE); \d+ ctlt12_storage - Table "public.ctlt12_storage" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+------+-----------+----------+---------+----------+--------------+------------- - a | text | | not null | | main | | - b | text | | | | extended | | - c | text | | | | external | | + Table "public.ctlt12_storage" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | not null | | main | pglz | | + b | text | | | | extended | pglz | | + c | text | | | | external | pglz | | CREATE TABLE ctlt12_comments (LIKE ctlt1 INCLUDING COMMENTS, LIKE ctlt2 INCLUDING COMMENTS); \d+ ctlt12_comments - Table "public.ctlt12_comments" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+------+-----------+----------+---------+----------+--------------+------------- - a | text | | not null | | extended | | A - b | text | | | | extended | | B - c | text | | | | extended | | C + Table "public.ctlt12_comments" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | not null | | extended | pglz | | A + b | text | | | | extended | pglz | | B + c | text | | | | extended | pglz | | C CREATE TABLE ctlt1_inh (LIKE ctlt1 INCLUDING CONSTRAINTS INCLUDING COMMENTS) INHERITS (ctlt1); NOTICE: merging column "a" with inherited definition NOTICE: merging column "b" with inherited definition NOTICE: merging constraint "ctlt1_a_check" with inherited definition \d+ ctlt1_inh - Table "public.ctlt1_inh" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+------+-----------+----------+---------+----------+--------------+------------- - a | text | | not null | | main | | A - b | text | | | | extended | | B + Table "public.ctlt1_inh" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | not null | | main | pglz | | A + b | text | | | | extended | pglz | | B Check constraints: "ctlt1_a_check" CHECK (length(a) > 2) Inherits: ctlt1 @@ -364,12 +364,12 @@ SELECT description FROM pg_description, pg_constraint c WHERE classoid = 'pg_con CREATE TABLE ctlt13_inh () INHERITS (ctlt1, ctlt3); NOTICE: merging multiple inherited definitions of column "a" \d+ ctlt13_inh - Table "public.ctlt13_inh" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+------+-----------+----------+---------+----------+--------------+------------- - a | text | | not null | | main | | - b | text | | | | extended | | - c | text | | | | external | | + Table "public.ctlt13_inh" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | not null | | main | pglz | | + b | text | | | | extended | pglz | | + c | text | | | | external | pglz | | Check constraints: "ctlt1_a_check" CHECK (length(a) > 2) "ctlt3_a_check" CHECK (length(a) < 5) @@ -380,12 +380,12 @@ Inherits: ctlt1, CREATE TABLE ctlt13_like (LIKE ctlt3 INCLUDING CONSTRAINTS INCLUDING INDEXES INCLUDING COMMENTS INCLUDING STORAGE) INHERITS (ctlt1); NOTICE: merging column "a" with inherited definition \d+ ctlt13_like - Table "public.ctlt13_like" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+------+-----------+----------+---------+----------+--------------+------------- - a | text | | not null | | main | | A3 - b | text | | | | extended | | - c | text | | | | external | | C + Table "public.ctlt13_like" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | not null | | main | pglz | | A3 + b | text | | | | extended | pglz | | + c | text | | | | external | pglz | | C Indexes: "ctlt13_like_expr_idx" btree ((a || c)) Check constraints: @@ -402,11 +402,11 @@ SELECT description FROM pg_description, pg_constraint c WHERE classoid = 'pg_con CREATE TABLE ctlt_all (LIKE ctlt1 INCLUDING ALL); \d+ ctlt_all - Table "public.ctlt_all" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+------+-----------+----------+---------+----------+--------------+------------- - a | text | | not null | | main | | A - b | text | | | | extended | | B + Table "public.ctlt_all" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | not null | | main | pglz | | A + b | text | | | | extended | pglz | | B Indexes: "ctlt_all_pkey" PRIMARY KEY, btree (a) "ctlt_all_b_idx" btree (b) @@ -440,11 +440,11 @@ DETAIL: MAIN versus EXTENDED -- Check that LIKE isn't confused by a system catalog of the same name CREATE TABLE pg_attrdef (LIKE ctlt1 INCLUDING ALL); \d+ public.pg_attrdef - Table "public.pg_attrdef" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+------+-----------+----------+---------+----------+--------------+------------- - a | text | | not null | | main | | A - b | text | | | | extended | | B + Table "public.pg_attrdef" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | not null | | main | pglz | | A + b | text | | | | extended | pglz | | B Indexes: "pg_attrdef_pkey" PRIMARY KEY, btree (a) "pg_attrdef_b_idx" btree (b) @@ -461,11 +461,11 @@ CREATE SCHEMA ctl_schema; SET LOCAL search_path = ctl_schema, public; CREATE TABLE ctlt1 (LIKE ctlt1 INCLUDING ALL); \d+ ctlt1 - Table "ctl_schema.ctlt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+------+-----------+----------+---------+----------+--------------+------------- - a | text | | not null | | main | | A - b | text | | | | extended | | B + Table "ctl_schema.ctlt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | not null | | main | pglz | | A + b | text | | | | extended | pglz | | B Indexes: "ctlt1_pkey" PRIMARY KEY, btree (a) "ctlt1_b_idx" btree (b) diff --git a/src/test/regress/expected/domain.out b/src/test/regress/expected/domain.out index 411d5c003e..6268b26562 100644 --- a/src/test/regress/expected/domain.out +++ b/src/test/regress/expected/domain.out @@ -266,10 +266,10 @@ explain (verbose, costs off) create rule silly as on delete to dcomptable do instead update dcomptable set d1.r = (d1).r - 1, d1.i = (d1).i + 1 where (d1).i > 0; \d+ dcomptable - Table "public.dcomptable" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+-----------+-----------+----------+---------+----------+--------------+------------- - d1 | dcomptype | | | | extended | | + Table "public.dcomptable" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-----------+-----------+----------+---------+----------+-------------+--------------+------------- + d1 | dcomptype | | | | extended | pglz | | Indexes: "dcomptable_d1_key" UNIQUE CONSTRAINT, btree (d1) Rules: @@ -403,10 +403,10 @@ create rule silly as on delete to dcomptable do instead update dcomptable set d1[1].r = d1[1].r - 1, d1[1].i = d1[1].i + 1 where d1[1].i > 0; \d+ dcomptable - Table "public.dcomptable" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+------------+-----------+----------+---------+----------+--------------+------------- - d1 | dcomptypea | | | | extended | | + Table "public.dcomptable" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------------+-----------+----------+---------+----------+-------------+--------------+------------- + d1 | dcomptypea | | | | extended | pglz | | Indexes: "dcomptable_d1_key" UNIQUE CONSTRAINT, btree (d1) Rules: diff --git a/src/test/regress/expected/foreign_data.out b/src/test/regress/expected/foreign_data.out index b9e25820bc..89466da603 100644 --- a/src/test/regress/expected/foreign_data.out +++ b/src/test/regress/expected/foreign_data.out @@ -1383,12 +1383,12 @@ CREATE TABLE fd_pt1 ( CREATE FOREIGN TABLE ft2 () INHERITS (fd_pt1) SERVER s0 OPTIONS (delimiter ',', quote '"', "be quoted" 'value'); \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Child tables: ft2 \d+ ft2 @@ -1404,12 +1404,12 @@ Inherits: fd_pt1 DROP FOREIGN TABLE ft2; \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | CREATE FOREIGN TABLE ft2 ( c1 integer NOT NULL, @@ -1428,12 +1428,12 @@ FDW options: (delimiter ',', quote '"', "be quoted" 'value') ALTER FOREIGN TABLE ft2 INHERIT fd_pt1; \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Child tables: ft2 \d+ ft2 @@ -1471,12 +1471,12 @@ Child tables: ct3, ft3 \d+ ct3 - Table "public.ct3" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Table "public.ct3" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Inherits: ft2 \d+ ft3 @@ -1496,17 +1496,17 @@ ALTER TABLE fd_pt1 ADD COLUMN c6 integer; ALTER TABLE fd_pt1 ADD COLUMN c7 integer NOT NULL; ALTER TABLE fd_pt1 ADD COLUMN c8 integer; \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | - c4 | integer | | | | plain | | - c5 | integer | | | 0 | plain | | - c6 | integer | | | | plain | | - c7 | integer | | not null | | plain | | - c8 | integer | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | + c4 | integer | | | | plain | | | + c5 | integer | | | 0 | plain | | | + c6 | integer | | | | plain | | | + c7 | integer | | not null | | plain | | | + c8 | integer | | | | plain | | | Child tables: ft2 \d+ ft2 @@ -1528,17 +1528,17 @@ Child tables: ct3, ft3 \d+ ct3 - Table "public.ct3" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | - c4 | integer | | | | plain | | - c5 | integer | | | 0 | plain | | - c6 | integer | | | | plain | | - c7 | integer | | not null | | plain | | - c8 | integer | | | | plain | | + Table "public.ct3" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | + c4 | integer | | | | plain | | | + c5 | integer | | | 0 | plain | | | + c6 | integer | | | | plain | | | + c7 | integer | | not null | | plain | | | + c8 | integer | | | | plain | | | Inherits: ft2 \d+ ft3 @@ -1570,17 +1570,17 @@ ALTER TABLE fd_pt1 ALTER COLUMN c1 SET (n_distinct = 100); ALTER TABLE fd_pt1 ALTER COLUMN c8 SET STATISTICS -1; ALTER TABLE fd_pt1 ALTER COLUMN c8 SET STORAGE EXTERNAL; \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | 10000 | - c2 | text | | | | extended | | - c3 | date | | | | plain | | - c4 | integer | | | 0 | plain | | - c5 | integer | | | | plain | | - c6 | integer | | not null | | plain | | - c7 | integer | | | | plain | | - c8 | text | | | | external | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | 10000 | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | + c4 | integer | | | 0 | plain | | | + c5 | integer | | | | plain | | | + c6 | integer | | not null | | plain | | | + c7 | integer | | | | plain | | | + c8 | text | | | | external | pglz | | Child tables: ft2 \d+ ft2 @@ -1608,12 +1608,12 @@ ALTER TABLE fd_pt1 DROP COLUMN c6; ALTER TABLE fd_pt1 DROP COLUMN c7; ALTER TABLE fd_pt1 DROP COLUMN c8; \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | 10000 | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | 10000 | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Child tables: ft2 \d+ ft2 @@ -1645,12 +1645,12 @@ SELECT relname, conname, contype, conislocal, coninhcount, connoinherit -- child does not inherit NO INHERIT constraints \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | 10000 | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | 10000 | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Check constraints: "fd_pt1chk1" CHECK (c1 > 0) NO INHERIT "fd_pt1chk2" CHECK (c2 <> ''::text) @@ -1692,12 +1692,12 @@ ALTER FOREIGN TABLE ft2 ADD CONSTRAINT fd_pt1chk2 CHECK (c2 <> ''); ALTER FOREIGN TABLE ft2 INHERIT fd_pt1; -- child does not inherit NO INHERIT constraints \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | 10000 | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | 10000 | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Check constraints: "fd_pt1chk1" CHECK (c1 > 0) NO INHERIT "fd_pt1chk2" CHECK (c2 <> ''::text) @@ -1723,12 +1723,12 @@ ALTER TABLE fd_pt1 DROP CONSTRAINT fd_pt1chk2 CASCADE; INSERT INTO fd_pt1 VALUES (1, 'fd_pt1'::text, '1994-01-01'::date); ALTER TABLE fd_pt1 ADD CONSTRAINT fd_pt1chk3 CHECK (c2 <> '') NOT VALID; \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | 10000 | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | 10000 | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Check constraints: "fd_pt1chk3" CHECK (c2 <> ''::text) NOT VALID Child tables: ft2 @@ -1750,12 +1750,12 @@ Inherits: fd_pt1 -- VALIDATE CONSTRAINT need do nothing on foreign tables ALTER TABLE fd_pt1 VALIDATE CONSTRAINT fd_pt1chk3; \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | 10000 | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | 10000 | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Check constraints: "fd_pt1chk3" CHECK (c2 <> ''::text) Child tables: ft2 @@ -1781,12 +1781,12 @@ ALTER TABLE fd_pt1 RENAME COLUMN c3 TO f3; -- changes name of a constraint recursively ALTER TABLE fd_pt1 RENAME CONSTRAINT fd_pt1chk3 TO f2_check; \d+ fd_pt1 - Table "public.fd_pt1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - f1 | integer | | not null | | plain | 10000 | - f2 | text | | | | extended | | - f3 | date | | | | plain | | + Table "public.fd_pt1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | integer | | not null | | plain | | 10000 | + f2 | text | | | | extended | pglz | | + f3 | date | | | | plain | | | Check constraints: "f2_check" CHECK (f2 <> ''::text) Child tables: ft2 @@ -1845,12 +1845,12 @@ CREATE TABLE fd_pt2 ( CREATE FOREIGN TABLE fd_pt2_1 PARTITION OF fd_pt2 FOR VALUES IN (1) SERVER s0 OPTIONS (delimiter ',', quote '"', "be quoted" 'value'); \d+ fd_pt2 - Partitioned table "public.fd_pt2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Partitioned table "public.fd_pt2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Partition key: LIST (c1) Partitions: fd_pt2_1 FOR VALUES IN (1) @@ -1890,12 +1890,12 @@ ERROR: table "fd_pt2_1" contains column "c4" not found in parent "fd_pt2" DETAIL: The new partition may contain only the columns present in parent. DROP FOREIGN TABLE fd_pt2_1; \d+ fd_pt2 - Partitioned table "public.fd_pt2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Partitioned table "public.fd_pt2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Partition key: LIST (c1) Number of partitions: 0 @@ -1917,12 +1917,12 @@ FDW options: (delimiter ',', quote '"', "be quoted" 'value') -- no attach partition validation occurs for foreign tables ALTER TABLE fd_pt2 ATTACH PARTITION fd_pt2_1 FOR VALUES IN (1); \d+ fd_pt2 - Partitioned table "public.fd_pt2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Partitioned table "public.fd_pt2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Partition key: LIST (c1) Partitions: fd_pt2_1 FOR VALUES IN (1) @@ -1945,12 +1945,12 @@ ERROR: cannot add column to a partition ALTER TABLE fd_pt2_1 ALTER c3 SET NOT NULL; ALTER TABLE fd_pt2_1 ADD CONSTRAINT p21chk CHECK (c2 <> ''); \d+ fd_pt2 - Partitioned table "public.fd_pt2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | | | extended | | - c3 | date | | | | plain | | + Partitioned table "public.fd_pt2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | | | extended | pglz | | + c3 | date | | | | plain | | | Partition key: LIST (c1) Partitions: fd_pt2_1 FOR VALUES IN (1) @@ -1975,12 +1975,12 @@ ERROR: column "c1" is marked NOT NULL in parent table ALTER TABLE fd_pt2 DETACH PARTITION fd_pt2_1; ALTER TABLE fd_pt2 ALTER c2 SET NOT NULL; \d+ fd_pt2 - Partitioned table "public.fd_pt2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | not null | | extended | | - c3 | date | | | | plain | | + Partitioned table "public.fd_pt2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | not null | | extended | pglz | | + c3 | date | | | | plain | | | Partition key: LIST (c1) Number of partitions: 0 @@ -2003,12 +2003,12 @@ ALTER TABLE fd_pt2 ATTACH PARTITION fd_pt2_1 FOR VALUES IN (1); ALTER TABLE fd_pt2 DETACH PARTITION fd_pt2_1; ALTER TABLE fd_pt2 ADD CONSTRAINT fd_pt2chk1 CHECK (c1 > 0); \d+ fd_pt2 - Partitioned table "public.fd_pt2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - c1 | integer | | not null | | plain | | - c2 | text | | not null | | extended | | - c3 | date | | | | plain | | + Partitioned table "public.fd_pt2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + c1 | integer | | not null | | plain | | | + c2 | text | | not null | | extended | pglz | | + c3 | date | | | | plain | | | Partition key: LIST (c1) Check constraints: "fd_pt2chk1" CHECK (c1 > 0) diff --git a/src/test/regress/expected/identity.out b/src/test/regress/expected/identity.out index fbca0333a2..dc2af075ff 100644 --- a/src/test/regress/expected/identity.out +++ b/src/test/regress/expected/identity.out @@ -498,14 +498,14 @@ TABLE itest8; (2 rows) \d+ itest8 - Table "public.itest8" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+----------------------------------+---------+--------------+------------- - f1 | integer | | | | plain | | - f2 | integer | | not null | generated always as identity | plain | | - f3 | integer | | not null | generated by default as identity | plain | | - f4 | bigint | | not null | generated always as identity | plain | | - f5 | bigint | | | | plain | | + Table "public.itest8" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+----------------------------------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + f2 | integer | | not null | generated always as identity | plain | | | + f3 | integer | | not null | generated by default as identity | plain | | | + f4 | bigint | | not null | generated always as identity | plain | | | + f5 | bigint | | | | plain | | | \d itest8_f2_seq Sequence "public.itest8_f2_seq" diff --git a/src/test/regress/expected/inherit.out b/src/test/regress/expected/inherit.out index 2b68aef654..7e709916f6 100644 --- a/src/test/regress/expected/inherit.out +++ b/src/test/regress/expected/inherit.out @@ -1052,13 +1052,13 @@ ALTER TABLE inhts RENAME aa TO aaa; -- to be failed ERROR: cannot rename inherited column "aa" ALTER TABLE inhts RENAME d TO dd; \d+ inhts - Table "public.inhts" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - aa | integer | | | | plain | | - b | integer | | | | plain | | - c | integer | | | | plain | | - dd | integer | | | | plain | | + Table "public.inhts" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + aa | integer | | | | plain | | | + b | integer | | | | plain | | | + c | integer | | | | plain | | | + dd | integer | | | | plain | | | Inherits: inht1, inhs1 @@ -1071,14 +1071,14 @@ NOTICE: merging multiple inherited definitions of column "aa" NOTICE: merging multiple inherited definitions of column "b" ALTER TABLE inht1 RENAME aa TO aaa; \d+ inht4 - Table "public.inht4" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - aaa | integer | | | | plain | | - b | integer | | | | plain | | - x | integer | | | | plain | | - y | integer | | | | plain | | - z | integer | | | | plain | | + Table "public.inht4" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + aaa | integer | | | | plain | | | + b | integer | | | | plain | | | + x | integer | | | | plain | | | + y | integer | | | | plain | | | + z | integer | | | | plain | | | Inherits: inht2, inht3 @@ -1088,14 +1088,14 @@ ALTER TABLE inht1 RENAME aaa TO aaaa; ALTER TABLE inht1 RENAME b TO bb; -- to be failed ERROR: cannot rename inherited column "b" \d+ inhts - Table "public.inhts" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - aaaa | integer | | | | plain | | - b | integer | | | | plain | | - x | integer | | | | plain | | - c | integer | | | | plain | | - d | integer | | | | plain | | + Table "public.inhts" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + aaaa | integer | | | | plain | | | + b | integer | | | | plain | | | + x | integer | | | | plain | | | + c | integer | | | | plain | | | + d | integer | | | | plain | | | Inherits: inht2, inhs1 @@ -1135,33 +1135,33 @@ drop cascades to table inht4 CREATE TABLE test_constraints (id int, val1 varchar, val2 int, UNIQUE(val1, val2)); CREATE TABLE test_constraints_inh () INHERITS (test_constraints); \d+ test_constraints - Table "public.test_constraints" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+-------------------+-----------+----------+---------+----------+--------------+------------- - id | integer | | | | plain | | - val1 | character varying | | | | extended | | - val2 | integer | | | | plain | | + Table "public.test_constraints" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + id | integer | | | | plain | | | + val1 | character varying | | | | extended | pglz | | + val2 | integer | | | | plain | | | Indexes: "test_constraints_val1_val2_key" UNIQUE CONSTRAINT, btree (val1, val2) Child tables: test_constraints_inh ALTER TABLE ONLY test_constraints DROP CONSTRAINT test_constraints_val1_val2_key; \d+ test_constraints - Table "public.test_constraints" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+-------------------+-----------+----------+---------+----------+--------------+------------- - id | integer | | | | plain | | - val1 | character varying | | | | extended | | - val2 | integer | | | | plain | | + Table "public.test_constraints" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + id | integer | | | | plain | | | + val1 | character varying | | | | extended | pglz | | + val2 | integer | | | | plain | | | Child tables: test_constraints_inh \d+ test_constraints_inh - Table "public.test_constraints_inh" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+-------------------+-----------+----------+---------+----------+--------------+------------- - id | integer | | | | plain | | - val1 | character varying | | | | extended | | - val2 | integer | | | | plain | | + Table "public.test_constraints_inh" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + id | integer | | | | plain | | | + val1 | character varying | | | | extended | pglz | | + val2 | integer | | | | plain | | | Inherits: test_constraints DROP TABLE test_constraints_inh; @@ -1172,27 +1172,27 @@ CREATE TABLE test_ex_constraints ( ); CREATE TABLE test_ex_constraints_inh () INHERITS (test_ex_constraints); \d+ test_ex_constraints - Table "public.test_ex_constraints" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+--------+-----------+----------+---------+---------+--------------+------------- - c | circle | | | | plain | | + Table "public.test_ex_constraints" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+--------+-----------+----------+---------+---------+-------------+--------------+------------- + c | circle | | | | plain | | | Indexes: "test_ex_constraints_c_excl" EXCLUDE USING gist (c WITH &&) Child tables: test_ex_constraints_inh ALTER TABLE test_ex_constraints DROP CONSTRAINT test_ex_constraints_c_excl; \d+ test_ex_constraints - Table "public.test_ex_constraints" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+--------+-----------+----------+---------+---------+--------------+------------- - c | circle | | | | plain | | + Table "public.test_ex_constraints" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+--------+-----------+----------+---------+---------+-------------+--------------+------------- + c | circle | | | | plain | | | Child tables: test_ex_constraints_inh \d+ test_ex_constraints_inh - Table "public.test_ex_constraints_inh" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+--------+-----------+----------+---------+---------+--------------+------------- - c | circle | | | | plain | | + Table "public.test_ex_constraints_inh" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+--------+-----------+----------+---------+---------+-------------+--------------+------------- + c | circle | | | | plain | | | Inherits: test_ex_constraints DROP TABLE test_ex_constraints_inh; @@ -1202,37 +1202,37 @@ CREATE TABLE test_primary_constraints(id int PRIMARY KEY); CREATE TABLE test_foreign_constraints(id1 int REFERENCES test_primary_constraints(id)); CREATE TABLE test_foreign_constraints_inh () INHERITS (test_foreign_constraints); \d+ test_primary_constraints - Table "public.test_primary_constraints" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - id | integer | | not null | | plain | | + Table "public.test_primary_constraints" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + id | integer | | not null | | plain | | | Indexes: "test_primary_constraints_pkey" PRIMARY KEY, btree (id) Referenced by: TABLE "test_foreign_constraints" CONSTRAINT "test_foreign_constraints_id1_fkey" FOREIGN KEY (id1) REFERENCES test_primary_constraints(id) \d+ test_foreign_constraints - Table "public.test_foreign_constraints" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - id1 | integer | | | | plain | | + Table "public.test_foreign_constraints" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + id1 | integer | | | | plain | | | Foreign-key constraints: "test_foreign_constraints_id1_fkey" FOREIGN KEY (id1) REFERENCES test_primary_constraints(id) Child tables: test_foreign_constraints_inh ALTER TABLE test_foreign_constraints DROP CONSTRAINT test_foreign_constraints_id1_fkey; \d+ test_foreign_constraints - Table "public.test_foreign_constraints" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - id1 | integer | | | | plain | | + Table "public.test_foreign_constraints" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + id1 | integer | | | | plain | | | Child tables: test_foreign_constraints_inh \d+ test_foreign_constraints_inh - Table "public.test_foreign_constraints_inh" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - id1 | integer | | | | plain | | + Table "public.test_foreign_constraints_inh" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + id1 | integer | | | | plain | | | Inherits: test_foreign_constraints DROP TABLE test_foreign_constraints_inh; diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index da50ee3b67..3cd4ad9e53 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -142,11 +142,11 @@ create rule irule3 as on insert to inserttest2 do also insert into inserttest (f4[1].if1, f4[1].if2[2]) select new.f1, new.f2; \d+ inserttest2 - Table "public.inserttest2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+--------+-----------+----------+---------+----------+--------------+------------- - f1 | bigint | | | | plain | | - f2 | text | | | | extended | | + Table "public.inserttest2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+--------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | bigint | | | | plain | | | + f2 | text | | | | extended | pglz | | Rules: irule1 AS ON INSERT TO inserttest2 DO INSERT INTO inserttest (f3.if2[1], f3.if2[2]) @@ -448,11 +448,11 @@ from hash_parted order by part; -- test \d+ output on a table which has both partitioned and unpartitioned -- partitions \d+ list_parted - Partitioned table "public.list_parted" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Partitioned table "public.list_parted" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition key: LIST (lower(a)) Partitions: part_aa_bb FOR VALUES IN ('aa', 'bb'), part_cc_dd FOR VALUES IN ('cc', 'dd'), @@ -470,10 +470,10 @@ drop table hash_parted; create table list_parted (a int) partition by list (a); create table part_default partition of list_parted default; \d+ part_default - Table "public.part_default" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | + Table "public.part_default" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | Partition of: list_parted DEFAULT No partition constraint @@ -853,11 +853,11 @@ create table mcrparted6_common_ge_10 partition of mcrparted for values from ('co create table mcrparted7_gt_common_lt_d partition of mcrparted for values from ('common', maxvalue) to ('d', minvalue); create table mcrparted8_ge_d partition of mcrparted for values from ('d', minvalue) to (maxvalue, maxvalue); \d+ mcrparted - Partitioned table "public.mcrparted" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Partitioned table "public.mcrparted" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition key: RANGE (a, b) Partitions: mcrparted1_lt_b FOR VALUES FROM (MINVALUE, MINVALUE) TO ('b', MINVALUE), mcrparted2_b FOR VALUES FROM ('b', MINVALUE) TO ('c', MINVALUE), @@ -869,74 +869,74 @@ Partitions: mcrparted1_lt_b FOR VALUES FROM (MINVALUE, MINVALUE) TO ('b', MINVAL mcrparted8_ge_d FOR VALUES FROM ('d', MINVALUE) TO (MAXVALUE, MAXVALUE) \d+ mcrparted1_lt_b - Table "public.mcrparted1_lt_b" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Table "public.mcrparted1_lt_b" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition of: mcrparted FOR VALUES FROM (MINVALUE, MINVALUE) TO ('b', MINVALUE) Partition constraint: ((a IS NOT NULL) AND (b IS NOT NULL) AND (a < 'b'::text)) \d+ mcrparted2_b - Table "public.mcrparted2_b" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Table "public.mcrparted2_b" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition of: mcrparted FOR VALUES FROM ('b', MINVALUE) TO ('c', MINVALUE) Partition constraint: ((a IS NOT NULL) AND (b IS NOT NULL) AND (a >= 'b'::text) AND (a < 'c'::text)) \d+ mcrparted3_c_to_common - Table "public.mcrparted3_c_to_common" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Table "public.mcrparted3_c_to_common" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition of: mcrparted FOR VALUES FROM ('c', MINVALUE) TO ('common', MINVALUE) Partition constraint: ((a IS NOT NULL) AND (b IS NOT NULL) AND (a >= 'c'::text) AND (a < 'common'::text)) \d+ mcrparted4_common_lt_0 - Table "public.mcrparted4_common_lt_0" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Table "public.mcrparted4_common_lt_0" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition of: mcrparted FOR VALUES FROM ('common', MINVALUE) TO ('common', 0) Partition constraint: ((a IS NOT NULL) AND (b IS NOT NULL) AND (a = 'common'::text) AND (b < 0)) \d+ mcrparted5_common_0_to_10 - Table "public.mcrparted5_common_0_to_10" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Table "public.mcrparted5_common_0_to_10" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition of: mcrparted FOR VALUES FROM ('common', 0) TO ('common', 10) Partition constraint: ((a IS NOT NULL) AND (b IS NOT NULL) AND (a = 'common'::text) AND (b >= 0) AND (b < 10)) \d+ mcrparted6_common_ge_10 - Table "public.mcrparted6_common_ge_10" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Table "public.mcrparted6_common_ge_10" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition of: mcrparted FOR VALUES FROM ('common', 10) TO ('common', MAXVALUE) Partition constraint: ((a IS NOT NULL) AND (b IS NOT NULL) AND (a = 'common'::text) AND (b >= 10)) \d+ mcrparted7_gt_common_lt_d - Table "public.mcrparted7_gt_common_lt_d" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Table "public.mcrparted7_gt_common_lt_d" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition of: mcrparted FOR VALUES FROM ('common', MAXVALUE) TO ('d', MINVALUE) Partition constraint: ((a IS NOT NULL) AND (b IS NOT NULL) AND (a > 'common'::text) AND (a < 'd'::text)) \d+ mcrparted8_ge_d - Table "public.mcrparted8_ge_d" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | integer | | | | plain | | + Table "public.mcrparted8_ge_d" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | integer | | | | plain | | | Partition of: mcrparted FOR VALUES FROM ('d', MINVALUE) TO (MAXVALUE, MAXVALUE) Partition constraint: ((a IS NOT NULL) AND (b IS NOT NULL) AND (a >= 'd'::text)) diff --git a/src/test/regress/expected/matview.out b/src/test/regress/expected/matview.out index 4b3a2e0cb7..e078818496 100644 --- a/src/test/regress/expected/matview.out +++ b/src/test/regress/expected/matview.out @@ -94,11 +94,11 @@ CREATE MATERIALIZED VIEW mvtest_bb AS SELECT * FROM mvtest_tvvmv; CREATE INDEX mvtest_aa ON mvtest_bb (grandtot); -- check that plans seem reasonable \d+ mvtest_tvm - Materialized view "public.mvtest_tvm" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - type | text | | | | extended | | - totamt | numeric | | | | main | | + Materialized view "public.mvtest_tvm" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + type | text | | | | extended | pglz | | + totamt | numeric | | | | main | pglz | | View definition: SELECT mvtest_tv.type, mvtest_tv.totamt @@ -106,11 +106,11 @@ View definition: ORDER BY mvtest_tv.type; \d+ mvtest_tvm - Materialized view "public.mvtest_tvm" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - type | text | | | | extended | | - totamt | numeric | | | | main | | + Materialized view "public.mvtest_tvm" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + type | text | | | | extended | pglz | | + totamt | numeric | | | | main | pglz | | View definition: SELECT mvtest_tv.type, mvtest_tv.totamt @@ -118,19 +118,19 @@ View definition: ORDER BY mvtest_tv.type; \d+ mvtest_tvvm - Materialized view "public.mvtest_tvvm" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+---------+---------+--------------+------------- - grandtot | numeric | | | | main | | + Materialized view "public.mvtest_tvvm" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + grandtot | numeric | | | | main | pglz | | View definition: SELECT mvtest_tvv.grandtot FROM mvtest_tvv; \d+ mvtest_bb - Materialized view "public.mvtest_bb" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+---------+---------+--------------+------------- - grandtot | numeric | | | | main | | + Materialized view "public.mvtest_bb" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + grandtot | numeric | | | | main | pglz | | Indexes: "mvtest_aa" btree (grandtot) View definition: @@ -142,10 +142,10 @@ CREATE SCHEMA mvtest_mvschema; ALTER MATERIALIZED VIEW mvtest_tvm SET SCHEMA mvtest_mvschema; \d+ mvtest_tvm \d+ mvtest_tvmm - Materialized view "public.mvtest_tvmm" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+---------+---------+--------------+------------- - grandtot | numeric | | | | main | | + Materialized view "public.mvtest_tvmm" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + grandtot | numeric | | | | main | pglz | | Indexes: "mvtest_tvmm_expr" UNIQUE, btree ((grandtot > 0::numeric)) "mvtest_tvmm_pred" UNIQUE, btree (grandtot) WHERE grandtot < 0::numeric @@ -155,11 +155,11 @@ View definition: SET search_path = mvtest_mvschema, public; \d+ mvtest_tvm - Materialized view "mvtest_mvschema.mvtest_tvm" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - type | text | | | | extended | | - totamt | numeric | | | | main | | + Materialized view "mvtest_mvschema.mvtest_tvm" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + type | text | | | | extended | pglz | | + totamt | numeric | | | | main | pglz | | View definition: SELECT mvtest_tv.type, mvtest_tv.totamt @@ -356,11 +356,11 @@ UNION ALL CREATE MATERIALIZED VIEW mv_test2 AS SELECT moo, 2*moo FROM mvtest_vt2 UNION ALL SELECT moo, 3*moo FROM mvtest_vt2; \d+ mv_test2 - Materialized view "public.mv_test2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+---------+---------+--------------+------------- - moo | integer | | | | plain | | - ?column? | integer | | | | plain | | + Materialized view "public.mv_test2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + moo | integer | | | | plain | | | + ?column? | integer | | | | plain | | | View definition: SELECT mvtest_vt2.moo, 2 * mvtest_vt2.moo @@ -493,14 +493,14 @@ drop cascades to materialized view mvtest_mv_v_4 CREATE MATERIALIZED VIEW mv_unspecified_types AS SELECT 42 as i, 42.5 as num, 'foo' as u, 'foo'::unknown as u2, null as n; \d+ mv_unspecified_types - Materialized view "public.mv_unspecified_types" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+----------+--------------+------------- - i | integer | | | | plain | | - num | numeric | | | | main | | - u | text | | | | extended | | - u2 | text | | | | extended | | - n | text | | | | extended | | + Materialized view "public.mv_unspecified_types" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + i | integer | | | | plain | | | + num | numeric | | | | main | pglz | | + u | text | | | | extended | pglz | | + u2 | text | | | | extended | pglz | | + n | text | | | | extended | pglz | | View definition: SELECT 42 AS i, 42.5 AS num, diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..d6912f2109 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -2813,34 +2813,34 @@ CREATE TABLE tbl_heap(f1 int, f2 char(100)) using heap; CREATE VIEW view_heap_psql AS SELECT f1 from tbl_heap_psql; CREATE MATERIALIZED VIEW mat_view_heap_psql USING heap_psql AS SELECT f1 from tbl_heap_psql; \d+ tbl_heap_psql - Table "tableam_display.tbl_heap_psql" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+----------------+-----------+----------+---------+----------+--------------+------------- - f1 | integer | | | | plain | | - f2 | character(100) | | | | extended | | + Table "tableam_display.tbl_heap_psql" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+----------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + f2 | character(100) | | | | extended | pglz | | \d+ tbl_heap - Table "tableam_display.tbl_heap" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+----------------+-----------+----------+---------+----------+--------------+------------- - f1 | integer | | | | plain | | - f2 | character(100) | | | | extended | | + Table "tableam_display.tbl_heap" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+----------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + f2 | character(100) | | | | extended | pglz | | \set HIDE_TABLEAM off \d+ tbl_heap_psql - Table "tableam_display.tbl_heap_psql" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+----------------+-----------+----------+---------+----------+--------------+------------- - f1 | integer | | | | plain | | - f2 | character(100) | | | | extended | | + Table "tableam_display.tbl_heap_psql" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+----------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + f2 | character(100) | | | | extended | pglz | | Access method: heap_psql \d+ tbl_heap - Table "tableam_display.tbl_heap" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+----------------+-----------+----------+---------+----------+--------------+------------- - f1 | integer | | | | plain | | - f2 | character(100) | | | | extended | | + Table "tableam_display.tbl_heap" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+----------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + f2 | character(100) | | | | extended | pglz | | Access method: heap -- AM is displayed for tables, indexes and materialized views. @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out index 63d6ab7a4e..71388c197d 100644 --- a/src/test/regress/expected/publication.out +++ b/src/test/regress/expected/publication.out @@ -74,11 +74,11 @@ SELECT pubname, puballtables FROM pg_publication WHERE pubname = 'testpub_forall (1 row) \d+ testpub_tbl2 - Table "public.testpub_tbl2" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('testpub_tbl2_id_seq'::regclass) | plain | | - data | text | | | | extended | | + Table "public.testpub_tbl2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('testpub_tbl2_id_seq'::regclass) | plain | | | + data | text | | | | extended | pglz | | Indexes: "testpub_tbl2_pkey" PRIMARY KEY, btree (id) Publications: @@ -187,22 +187,22 @@ ALTER PUBLICATION testpub_default SET TABLE testpub_tbl1; ALTER PUBLICATION testpub_default ADD TABLE pub_test.testpub_nopk; ALTER PUBLICATION testpib_ins_trunct ADD TABLE pub_test.testpub_nopk, testpub_tbl1; \d+ pub_test.testpub_nopk - Table "pub_test.testpub_nopk" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - foo | integer | | | | plain | | - bar | integer | | | | plain | | + Table "pub_test.testpub_nopk" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + foo | integer | | | | plain | | | + bar | integer | | | | plain | | | Publications: "testpib_ins_trunct" "testpub_default" "testpub_fortbl" \d+ testpub_tbl1 - Table "public.testpub_tbl1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('testpub_tbl1_id_seq'::regclass) | plain | | - data | text | | | | extended | | + Table "public.testpub_tbl1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('testpub_tbl1_id_seq'::regclass) | plain | | | + data | text | | | | extended | pglz | | Indexes: "testpub_tbl1_pkey" PRIMARY KEY, btree (id) Publications: @@ -224,11 +224,11 @@ ALTER PUBLICATION testpub_default DROP TABLE testpub_tbl1, pub_test.testpub_nopk ALTER PUBLICATION testpub_default DROP TABLE pub_test.testpub_nopk; ERROR: relation "testpub_nopk" is not part of the publication \d+ testpub_tbl1 - Table "public.testpub_tbl1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('testpub_tbl1_id_seq'::regclass) | plain | | - data | text | | | | extended | | + Table "public.testpub_tbl1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('testpub_tbl1_id_seq'::regclass) | plain | | | + data | text | | | | extended | pglz | | Indexes: "testpub_tbl1_pkey" PRIMARY KEY, btree (id) Publications: diff --git a/src/test/regress/expected/replica_identity.out b/src/test/regress/expected/replica_identity.out index 79002197a7..20ac0012ef 100644 --- a/src/test/regress/expected/replica_identity.out +++ b/src/test/regress/expected/replica_identity.out @@ -153,13 +153,13 @@ SELECT relreplident FROM pg_class WHERE oid = 'test_replica_identity'::regclass; (1 row) \d+ test_replica_identity - Table "public.test_replica_identity" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('test_replica_identity_id_seq'::regclass) | plain | | - keya | text | | not null | | extended | | - keyb | text | | not null | | extended | | - nonkey | text | | | | extended | | + Table "public.test_replica_identity" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('test_replica_identity_id_seq'::regclass) | plain | | | + keya | text | | not null | | extended | pglz | | + keyb | text | | not null | | extended | pglz | | + nonkey | text | | | | extended | pglz | | Indexes: "test_replica_identity_pkey" PRIMARY KEY, btree (id) "test_replica_identity_expr" UNIQUE, btree (keya, keyb, (3)) diff --git a/src/test/regress/expected/rowsecurity.out b/src/test/regress/expected/rowsecurity.out index 9506aaef82..7ebdc30ded 100644 --- a/src/test/regress/expected/rowsecurity.out +++ b/src/test/regress/expected/rowsecurity.out @@ -938,14 +938,14 @@ CREATE POLICY pp1 ON part_document AS PERMISSIVE CREATE POLICY pp1r ON part_document AS RESTRICTIVE TO regress_rls_dave USING (cid < 55); \d+ part_document - Partitioned table "regress_rls_schema.part_document" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ----------+---------+-----------+----------+---------+----------+--------------+------------- - did | integer | | | | plain | | - cid | integer | | | | plain | | - dlevel | integer | | not null | | plain | | - dauthor | name | | | | plain | | - dtitle | text | | | | extended | | + Partitioned table "regress_rls_schema.part_document" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +---------+---------+-----------+----------+---------+----------+-------------+--------------+------------- + did | integer | | | | plain | | | + cid | integer | | | | plain | | | + dlevel | integer | | not null | | plain | | | + dauthor | name | | | | plain | | | + dtitle | text | | | | extended | pglz | | Partition key: RANGE (cid) Policies: POLICY "pp1" diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 6173473de9..8546f02c1c 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -3035,11 +3035,11 @@ select * from rules_log; create rule r3 as on delete to rules_src do notify rules_src_deletion; \d+ rules_src - Table "public.rules_src" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - f1 | integer | | | | plain | | - f2 | integer | | | | plain | | + Table "public.rules_src" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + f2 | integer | | | | plain | | | Rules: r1 AS ON UPDATE TO rules_src DO INSERT INTO rules_log (f1, f2, tag) VALUES (old.f1,old.f2,'old'::text), (new.f1,new.f2,'new'::text) @@ -3055,11 +3055,11 @@ Rules: create rule r4 as on insert to rules_src do instead insert into rules_log AS trgt SELECT NEW.* RETURNING trgt.f1, trgt.f2; create rule r5 as on update to rules_src do instead UPDATE rules_log AS trgt SET tag = 'updated' WHERE trgt.f1 = new.f1; \d+ rules_src - Table "public.rules_src" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - f1 | integer | | | | plain | | - f2 | integer | | | | plain | | + Table "public.rules_src" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + f2 | integer | | | | plain | | | Rules: r1 AS ON UPDATE TO rules_src DO INSERT INTO rules_log (f1, f2, tag) VALUES (old.f1,old.f2,'old'::text), (new.f1,new.f2,'new'::text) @@ -3086,11 +3086,11 @@ create rule rr as on update to rule_t1 do instead UPDATE rule_dest trgt SET (f2[1], f1, tag) = (SELECT new.f2, new.f1, 'updated'::varchar) WHERE trgt.f1 = new.f1 RETURNING new.*; \d+ rule_t1 - Table "public.rule_t1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - f1 | integer | | | | plain | | - f2 | integer | | | | plain | | + Table "public.rule_t1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + f2 | integer | | | | plain | | | Rules: rr AS ON UPDATE TO rule_t1 DO INSTEAD UPDATE rule_dest trgt SET (f2[1], f1, tag) = ( SELECT new.f2, diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out index 431b3fa3de..a36b7e6069 100644 --- a/src/test/regress/expected/stats_ext.out +++ b/src/test/regress/expected/stats_ext.out @@ -125,11 +125,11 @@ SELECT stxname, stxdndistinct, stxddependencies, stxdmcv ALTER STATISTICS ab1_a_b_stats SET STATISTICS -1; \d+ ab1 - Table "public.ab1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | - b | integer | | | | plain | | + Table "public.ab1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | + b | integer | | | | plain | | | Statistics objects: "public"."ab1_a_b_stats" (ndistinct, dependencies, mcv) ON a, b FROM ab1 diff --git a/src/test/regress/expected/update.out b/src/test/regress/expected/update.out index bf939d79f6..58e996fdee 100644 --- a/src/test/regress/expected/update.out +++ b/src/test/regress/expected/update.out @@ -711,14 +711,14 @@ DROP TRIGGER d15_insert_trig ON part_d_15_20; :init_range_parted; create table part_def partition of range_parted default; \d+ part_def - Table "public.part_def" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+-------------------+-----------+----------+---------+----------+--------------+------------- - a | text | | | | extended | | - b | bigint | | | | plain | | - c | numeric | | | | main | | - d | integer | | | | plain | | - e | character varying | | | | extended | | + Table "public.part_def" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + a | text | | | | extended | pglz | | + b | bigint | | | | plain | | | + c | numeric | | | | main | pglz | | + d | integer | | | | plain | | | + e | character varying | | | | extended | pglz | | Partition of: range_parted DEFAULT Partition constraint: (NOT ((a IS NOT NULL) AND (b IS NOT NULL) AND (((a = 'a'::text) AND (b >= '1'::bigint) AND (b < '10'::bigint)) OR ((a = 'a'::text) AND (b >= '10'::bigint) AND (b < '20'::bigint)) OR ((a = 'b'::text) AND (b >= '1'::bigint) AND (b < '10'::bigint)) OR ((a = 'b'::text) AND (b >= '10'::bigint) AND (b < '20'::bigint)) OR ((a = 'b'::text) AND (b >= '20'::bigint) AND (b < '30'::bigint))))) diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source index 94c5f023c6..458c297670 100644 --- a/src/test/regress/output/tablespace.source +++ b/src/test/regress/output/tablespace.source @@ -151,10 +151,10 @@ Indexes: Number of partitions: 2 (Use \d+ to list them.) \d+ testschema.part - Partitioned table "testschema.part" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | + Partitioned table "testschema.part" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | Partition key: LIST (a) Indexes: "part_a_idx" btree (a), tablespace "regress_tblspace" @@ -171,10 +171,10 @@ Indexes: "part1_a_idx" btree (a), tablespace "regress_tblspace" \d+ testschema.part1 - Table "testschema.part1" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description ---------+---------+-----------+----------+---------+---------+--------------+------------- - a | integer | | | | plain | | + Table "testschema.part1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + a | integer | | | | plain | | | Partition of: testschema.part FOR VALUES IN (1) Partition constraint: ((a IS NOT NULL) AND (a = 1)) Indexes: diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e0e1ef71dd..2407ca945b 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -116,6 +116,9 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +#test toast compression +test: compression + # event triggers cannot run concurrently with any test that runs DDL test: event_trigger # this test also uses event triggers, so likewise run it by itself diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 081fce32e7..152c8cb5b7 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -203,3 +203,4 @@ test: explain test: event_trigger test: fast_default test: stats +test: compression diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..0750a6c80e --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,74 @@ +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + + + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2f28de0355..9ebfe19a34 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 721b230bf2..49a62d6e33 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --ytoMbUMiTKPMT3hY Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v21-0002-Add-default_toast_compression-GUC.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 02/12] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..3374012940 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b4ab0b88ad..0db2a7430f 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index b2457a6924..811dbc3741 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..e262ec5bcb 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 1487710d59..4b3d34ae1a 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index 943931f65d..ac3884b3a8 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --9Ek0hoCL9XbhcSqy Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0003-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 01/10] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 62 files changed, 1746 insertions(+), 123 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v24 1/7] Built-in compression method @ 2021-02-05 12:51 dilipkumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 116 ++++++++++ configure.ac | 17 ++ contrib/test_decoding/expected/ddl.out | 50 ++--- doc/src/sgml/ddl.sgml | 3 + doc/src/sgml/ref/create_table.sgml | 25 +++ src/backend/access/Makefile | 2 +- src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/detoast.c | 103 ++++++--- src/backend/access/common/indextuple.c | 4 +- src/backend/access/common/toast_internals.c | 63 +++--- src/backend/access/common/tupdesc.c | 8 + src/backend/access/compression/Makefile | 17 ++ src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++ .../access/compression/compress_pglz.c | 138 ++++++++++++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 7 +- src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 2 + src/backend/catalog/toasting.c | 5 + src/backend/commands/amcmds.c | 10 + src/backend/commands/createas.c | 14 ++ src/backend/commands/matview.c | 14 ++ src/backend/commands/tablecmds.c | 111 +++++++++- src/backend/executor/nodeModifyTable.c | 122 +++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 ++- src/backend/parser/parse_utilcmd.c | 8 + src/backend/utils/adt/pseudotypes.c | 1 + src/backend/utils/adt/varlena.c | 42 ++++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 36 +++- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 28 ++- src/include/access/compressamapi.h | 99 +++++++++ src/include/access/detoast.h | 8 + src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 19 +- src/include/catalog/pg_am.dat | 6 + src/include/catalog/pg_am.h | 1 + src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 20 ++ src/include/catalog/pg_type.dat | 5 + src/include/commands/defrem.h | 1 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 12 +- src/test/regress/expected/compression.out | 201 ++++++++++++++++++ src/test/regress/expected/compression_1.out | 189 ++++++++++++++++ src/test/regress/expected/psql.out | 68 +++--- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 85 ++++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 63 files changed, 1771 insertions(+), 148 deletions(-) create mode 100644 src/backend/access/compression/Makefile create mode 100644 src/backend/access/compression/compress_lz4.c create mode 100644 src/backend/access/compression/compress_pglz.c create mode 100644 src/include/access/compressamapi.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index ce9ea36999..0895b2f326 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,39 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + + + # # Assignments # @@ -12054,6 +12090,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 07da84d401..63940b78a0 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out index 4ff0044c78..6ee077678d 100644 --- a/contrib/test_decoding/expected/ddl.out +++ b/contrib/test_decoding/expected/ddl.out @@ -438,12 +438,12 @@ CREATE TABLE replication_metadata ( WITH (user_catalog_table = true) ; \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('foo', ARRAY['a', 'b']); ALTER TABLE replication_metadata RESET (user_catalog_table); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) @@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options) VALUES ('bar', ARRAY['a', 'b']); ALTER TABLE replication_metadata SET (user_catalog_table = true); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=true @@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text; ERROR: cannot rewrite table "replication_metadata" used as a catalog table ALTER TABLE replication_metadata SET (user_catalog_table = false); \d+ replication_metadata - Table "public.replication_metadata" - Column | Type | Collation | Nullable | Default | Storage | Stats target | Description -----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+------------- - id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | - relation | name | | not null | | plain | | - options | text[] | | | | extended | | - rewritemeornot | integer | | | | plain | | + Table "public.replication_metadata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+------------- + id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | | + relation | name | | not null | | plain | | | + options | text[] | | | | extended | pglz | | + rewritemeornot | integer | | | | plain | | | Indexes: "replication_metadata_pkey" PRIMARY KEY, btree (id) Options: user_catalog_table=false diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 1e9a4625cc..4d7ed698b9 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -3762,6 +3762,9 @@ CREATE TABLE measurement ( <productname>PostgreSQL</productname> tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately. + By default, each column in a partition inherits the compression method + from parent table's column, however a different compression method can be + set for each partition. </para> <para> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 569f4c9da7..51a7a977a5 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] | UNIQUE <replaceable class="parameter">index_parameters</replaceable> | PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> | + COMPRESSION <replaceable class="parameter">compression_method</replaceable> | REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in the columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> @@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + This sets the compression method for a column. The supported compression + methods are <literal>pglz</literal> and <literal>lz4</literal>. + <literal>lz4</literal> is available only if <literal>--with-lz4</literal> + was used when building <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createtable-exclude"> <term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term> <listitem> diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile index 0880e0a8bb..ba08bdd63e 100644 --- a/src/backend/access/Makefile +++ b/src/backend/access/Makefile @@ -8,7 +8,7 @@ subdir = src/backend/access top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \ +SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \ table tablesample transam include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..b78d49167b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, } /* ---------- - * toast_decompress_datum - + * toast_get_compression_oid - * - * Decompress a compressed version of a varlena datum + * Returns the Oid of the compression method stored in the compressed data. If + * the varlena is not compressed then returns InvalidOid. */ -static struct varlena * -toast_decompress_datum(struct varlena *attr) +Oid +toast_get_compression_oid(struct varlena *attr) { - struct varlena *result; + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidOid; + + /* + * Just fetch the toast compress header to know the compression method + * in the compressed data. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidOid; + + return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr)); +} + +/* ---------- + * toast_get_compression_handler - get the compression handler routines + * + * helper function for toast_decompress_datum and toast_decompress_datum_slice + */ +static inline const CompressionAmRoutine * +toast_get_compression_handler(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine; + CompressionId cmid; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); + cmid = TOAST_COMPRESS_METHOD(attr); - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* Get the handler routines for the compression method */ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_ID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } - return result; + return cmroutine; +} + +/* ---------- + * toast_decompress_datum - + * + * Decompress a compressed version of a varlena datum + */ +static struct varlena * +toast_decompress_datum(struct varlena *attr) +{ + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); + + return cmroutine->datum_decompress(attr); } @@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; - - Assert(VARATT_IS_COMPRESSED(attr)); - - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + /* + * If the handler supports the slice decompression then decompress the + * slice otherwise decompress complete data. + */ + if (cmroutine->datum_decompress_slice) + return cmroutine->datum_decompress_slice(attr, slicelength); + else + return cmroutine->datum_decompress(attr); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1d43d5d2ff 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -16,6 +16,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/detoast.h" #include "access/heaptoast.h" #include "access/htup_details.h" @@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..b04c5a5eb8 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, Oid cmoid) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionAmRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(OidIsValid(cmoid)); + + /* Get the handler routines for the compression method */ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + cmroutine = &pglz_compress_methods; + break; + case LZ4_COMPRESSION_AM_OID: + cmroutine = &lz4_compress_methods; + break; + default: + elog(ERROR, "Invalid compression method oid %u", cmoid); + } - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* Call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..ca26fab487 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionOid; + else + att->attcompression = InvalidOid; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile new file mode 100644 index 0000000000..779f54e785 --- /dev/null +++ b/src/backend/access/compression/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for access/compression +# +# IDENTIFICATION +# src/backend/access/compression/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/access/compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = compress_pglz.o compress_lz4.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c new file mode 100644 index 0000000000..1856cf7df7 --- /dev/null +++ b/src/backend/access/compression/compress_lz4.c @@ -0,0 +1,164 @@ +/*------------------------------------------------------------------------- + * + * compress_lz4.c + * lz4 compression method. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/access/compression/compress_lz4.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/compressamapi.h" +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Get maximum size of the compressed data that lz4 compression may output + * and allocate the memory for holding the compressed data and the header. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress data using lz4 routine */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for holding the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress partial data using lz4 routine */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* ------------------------------------------------------------------------ + * Definition of the lz4 compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine lz4_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice +}; + +/* lz4 compression handler function */ +Datum +lz4handler(PG_FUNCTION_ARGS) +{ +#ifndef HAVE_LIBLZ4 + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#else + PG_RETURN_POINTER(&lz4_compress_methods); +#endif +} diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c new file mode 100644 index 0000000000..8a4bf427cf --- /dev/null +++ b/src/backend/access/compression/compress_pglz.c @@ -0,0 +1,138 @@ +/*------------------------------------------------------------------------- + * + * compress_pglz.c + * pglz compression method + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/compression/compress_pglz.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/compressamapi.h" +#include "common/pg_lzcompress.h" + +#include "fmgr.h" +#include "utils/builtins.h" + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Get maximum size of the compressed data that pglz compression may output + * and allocate the memory for holding the compressed data and the header. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* ------------------------------------------------------------------------ + * Definition of the pglz compression access method. + * ------------------------------------------------------------------------ + */ +const CompressionAmRoutine pglz_compress_methods = { + .type = T_CompressionAmRoutine, + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice +}; + +/* pglz compression handler function */ +Datum +pglzhandler(PG_FUNCTION_ARGS) +{ + PG_RETURN_POINTER(&pglz_compress_methods); +} diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..9b451eaa71 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionOid; + else + attrtypes[attnum]->attcompression = InvalidOid; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..f5bb37b972 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId; my $C_COLLATION_OID = Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation}, 'C_COLLATION_OID'); - +my $PGLZ_COMPRESSION_AM_OID = + Catalog::FindDefinedSymbolFromData($catalog_data{pg_am}, + 'PGLZ_COMPRESSION_AM_OID'); # Fill in pg_class.relnatts by looking at the referenced catalog's schema. # This is ugly but there's no better place; Catalog::AddDefaultValues @@ -906,6 +908,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..b53b6b50e6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidOid; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1514937748..d1d6bdf470 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -24,6 +24,7 @@ #include <unistd.h> #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/heapam.h" #include "access/multixact.h" #include "access/reloptions.h" @@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..a549481557 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index eff9535ed0..3ad4a61739 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok) return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok); } +/* + * get_compression_am_oid - given an access method name, look up its OID + * and verify it corresponds to an compression AM. + */ +Oid +get_compression_am_oid(const char *amname, bool missing_ok) +{ + return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok); +} + /* * get_am_oid - given an access method name, look up its OID. * The type is not checked. diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index dce882012e..1d17dc0d6b 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -61,6 +61,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_intorel; /* utility functions for CTAS definition creation */ @@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self) /* Nothing to insert if WITH NO DATA is specified. */ if (!myState->into->skipData) { + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->rel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly @@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self) /* close rel, but keep lock until commit */ table_close(myState->rel, NoLock); myState->rel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index c5c25ce11d..713fc3fceb 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -56,6 +56,7 @@ typedef struct CommandId output_cid; /* cmin to insert in output tuples */ int ti_options; /* table_tuple_insert performance options */ BulkInsertState bistate; /* bulk insert state */ + TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */ } DR_transientrel; static int matview_maintenance_depth = 0; @@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self) { DR_transientrel *myState = (DR_transientrel *) self; + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &myState->decompress_tuple_slot, + myState->transientrel->rd_att); + /* * Note that the input slot might not be of the type of the target * relation. That's supported by table_tuple_insert(), but slightly less @@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self) /* close transientrel, but keep lock until commit */ table_close(myState->transientrel, NoLock); myState->transientrel = NULL; + + /* release the slot used for decompressing the tuple */ + if (myState->decompress_tuple_slot) + ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 420991e315..dd81d5bf4e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/compressamapi.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); - +static Oid GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- * DefineRelation @@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidOid; } /* @@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (OidIsValid(attribute->attcompression)) + { + char *compression = get_am_name(attribute->attcompression); + + if (!def->compression) + def->compression = compression; + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + def->compression = get_am_name(attribute->attcompression); inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (!def->compression) + def->compression = newdef->compression; + else if (newdef->compression) + { + if (strcmp(def->compression, newdef->compression)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store its Oid in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidOid; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * InvalidOid for the plain/external storage otherwise default + * compression id. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidOid; + else if (!OidIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionOid; + } + else + attTup->attcompression = InvalidOid; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to an OID. + */ +static Oid +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + Oid amoid; + + /* + * No compression for the plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidOid; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionOid; + + amoid = get_compression_am_oid(compression, false); + +#ifndef HAVE_LIBLZ4 + if (amoid == LZ4_COMPRESSION_AM_OID) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("not built with lz4 support"))); +#endif + return amoid; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..f77deee399 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,6 +37,8 @@ #include "postgres.h" +#include "access/compressamapi.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" @@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not supported in the target attribute then + * decompress the value. If any of the value need to decompress then we + * need to store that into the new slot. + * + * The slot will hold the input slot but if any of the value need to be + * decompressed then the new slot will be stored into this and the old + * slot will be dropped. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + Oid cmoid; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method Oid stored in the toast header and + * compare it with the compression method of the target. + */ + cmoid = toast_get_compression_oid(new_value); + if (OidIsValid(cmoid) && + targetTupDesc->attrs[i].attcompression != cmoid) + { + new_value = detoast_attr(new_value); + slot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then we need to copy the + * values/null array from oldslot to the new slot and materialize the new + * slot. + */ + if (decompressed_any) + { + TupleTableSlot *newslot = *outslot; + + /* + * If the called has passed an invalid slot then create a new slot. + * Otherwise, just clear the existing tuple from the slot. This slot + * should be stored by the caller so that it can be reused for + * decompressing the subsequent tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops); + else + ExecClearTuple(newslot); + + /* + * Copy the value/null array to the new slot and materialize it, + * before clearing the tuple from the old slot. + */ + memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool)); + ExecStoreVirtualTuple(newslot); + ExecMaterializeSlot(newslot); + ExecClearTuple(slot); + + *outslot = newslot; + + return newslot; + } + + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 65bbc18ecb..1338e04409 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index f5dcedf6e8..0605ef3f84 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index dd72a9fc3c..52d92df25d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15285,6 +15297,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15805,6 +15818,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b31f3afa03..cf4413da64 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && OidIsValid(attribute->attcompression)) + def->compression = get_am_name(attribute->attcompression); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index c2f910d606..fe133c76c2 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..a35abe5f1a 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,9 +17,11 @@ #include <ctype.h> #include <limits.h> +#include "access/compressamapi.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" +#include "commands/defrem.h" #include "common/hashfn.h" #include "common/hex.h" #include "common/int.h" @@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or the uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + Oid cmoid; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + cmoid = + toast_get_compression_oid((struct varlena *) DatumGetPointer(value)); + + if (!OidIsValid(cmoid)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..7332f93a25 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_compression_methods; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..46044cb92a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-compression-methods do not dump compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "am.amname AS attcmname,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* need left join here to not fail on dropped columns ... */ appendPQExpBuffer(q, "FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t " - "ON a.atttypid = t.oid\n" + "ON a.atttypid = t.oid\n"); + + if (createWithCompression) + appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am " + "ON a.attcompression = am.oid\n"); + appendPQExpBuffer(q, "WHERE a.attrelid = '%u'::pg_catalog.oid " "AND a.attnum > 0::pg_catalog.int2\n" "ORDER BY a.attnum", @@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) { bool print_default; bool print_notnull; + bool has_non_default_compression; /* * Default value --- suppress if to be printed separately. @@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) !dopt->binary_upgrade) continue; + has_non_default_compression = (tbinfo->attcmnames[j] && + (strcmp(tbinfo->attcmnames[j], "pglz") != 0)); + /* Format properly if not first attr */ if (actual_atts == 0) appendPQExpBufferStr(q, " ("); @@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_compression_methods && + tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) && + (has_non_default_compression || dopt->binary_upgrade)) + { + appendPQExpBuffer(q, " COMPRESSION %s", + tbinfo->attcmnames[j]); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..1789e18f46 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char **attcmnames; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..97791f8dbe 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text\E\D*,\n + \s+\Qcol3 text\E\D*,\n + \s+\Qcol4 text\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5)\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..ba464d463e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose) " CASE amtype" " WHEN 'i' THEN '%s'" " WHEN 't' THEN '%s'" + " WHEN 'c' THEN '%s'" " END AS \"%s\"", gettext_noop("Name"), gettext_noop("Index"), gettext_noop("Table"), + gettext_noop("Compression"), gettext_noop("Type")); if (verbose) @@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE " + " (SELECT am.amname " + " FROM pg_catalog.pg_am am " + " WHERE am.oid = a.attcompression) " + " END AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression */ + if (attcompression_col >= 0) + printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col), + false, false); + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h new file mode 100644 index 0000000000..5a8e23d926 --- /dev/null +++ b/src/include/access/compressamapi.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * compressamapi.h + * API for Postgres compression methods. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/compressamapi.h + * + *------------------------------------------------------------------------- + */ + +#ifndef COMPRESSAMAPI_H +#define COMPRESSAMAPI_H + +#include "postgres.h" + +#include "catalog/pg_am_d.h" +#include "nodes/nodes.h" + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression method oid. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* Use default compression method if it is not specified. */ +#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression AM. + * + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionAmRoutine +{ + NodeTag type; + + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionAmRoutine; + +extern const CompressionAmRoutine pglz_compress_methods; +extern const CompressionAmRoutine lz4_compress_methods; + +/* + * CompressionOidToId - Convert compression Oid to built-in compression id. + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline CompressionId +CompressionOidToId(Oid cmoid) +{ + switch (cmoid) + { + case PGLZ_COMPRESSION_AM_OID: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION_AM_OID: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method oid %u", cmoid); + } +} + +/* + * CompressionIdToOid - Convert built-in compression id to Oid + * + * For more details refer comment atop CompressionId in compressamapi.h + */ +static inline Oid +CompressionIdToOid(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION_AM_OID; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION_AM_OID; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +#endif /* COMPRESSAMAPI_H */ diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..6cdc37542d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_oid - + * + * Return the compression method oid from the compressed value + * ---------- + */ +extern Oid toast_get_compression_oid(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..dca0bc37f3 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + Oid tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..31ff91a09c 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/compressamapi.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,22 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 info; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, Oid cmoid); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat index 6082f0e6a8..605684408c 100644 --- a/src/include/catalog/pg_am.dat +++ b/src/include/catalog/pg_am.dat @@ -33,5 +33,11 @@ { oid => '3580', oid_symbol => 'BRIN_AM_OID', descr => 'block range index (BRIN) access method', amname => 'brin', amhandler => 'brinhandler', amtype => 'i' }, +{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID', + descr => 'pglz compression access method', + amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' }, +{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID', + descr => 'lz4 compression access method', + amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' }, ] diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h index ced86faef8..65079f3821 100644 --- a/src/include/catalog/pg_am.h +++ b/src/include/catalog/pg_am.h @@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op */ #define AMTYPE_INDEX 'i' /* index access method */ #define AMTYPE_TABLE 't' /* table access method */ +#define AMTYPE_COMPRESSION 'c' /* compression access method */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..797e78b17c 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * OID of compression AM. Must be InvalidOid if and only if typstorage is + * 'plain' or 'external'. + */ + Oid attcompression BKI_DEFAULT(0); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4e0c9be58c..4a13844ce6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -941,6 +941,15 @@ prorettype => 'void', proargtypes => 'regclass int8', prosrc => 'brin_desummarize_range' }, +{ oid => '6015', descr => 'pglz compression access method handler', + proname => 'pglzhandler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'pglzhandler' }, +{ oid => '6016', descr => 'lz4 compression access method handler', + proname => 'lz4handler', provolatile => 'v', + prorettype => 'compression_am_handler', proargtypes => 'internal', + prosrc => 'lz4handler' }, + { oid => '338', descr => 'validate an operator class', proname => 'amvalidate', provolatile => 'v', prorettype => 'bool', proargtypes => 'oid', prosrc => 'amvalidate' }, @@ -7095,6 +7104,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', @@ -7265,6 +7278,13 @@ { oid => '268', descr => 'I/O', proname => 'table_am_handler_out', prorettype => 'cstring', proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' }, +{ oid => '560', descr => 'I/O', + proname => 'compression_am_handler_in', proisstrict => 'f', + prorettype => 'compression_am_handler', proargtypes => 'cstring', + prosrc => 'compression_am_handler_in' }, +{ oid => '561', descr => 'I/O', + proname => 'compression_am_handler_out', prorettype => 'cstring', + proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' }, { oid => '5086', descr => 'I/O', proname => 'anycompatible_in', prorettype => 'anycompatible', proargtypes => 'cstring', prosrc => 'anycompatible_in' }, diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 8959c2f53b..306aabf6c1 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -626,6 +626,11 @@ typcategory => 'P', typinput => 'index_am_handler_in', typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, + { oid => '5559', + typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'compression_am_handler_in', + typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '3310', descr => 'pseudo-type for the result of a tablesample method function', typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index 1a79540c94..e5aea8a240 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId, extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt); extern Oid get_index_am_oid(const char *amname, bool missing_ok); extern Oid get_table_am_oid(const char *amname, bool missing_ok); +extern Oid get_compression_am_oid(const char *amname, bool missing_ok); extern Oid get_am_oid(const char *amname, bool missing_ok); extern char *get_am_name(Oid amOid); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index b6a88ff76b..18e8ecfd90 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1182,6 +1182,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, incase the target attribute's + * compression method doesn't match with the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 40ae489c23..20d6f96f62 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -512,6 +512,7 @@ typedef enum NodeTag T_IndexAmRoutine, /* in access/amapi.h */ T_TableAmRoutine, /* in access/tableam.h */ T_TsmRoutine, /* in access/tsmapi.h */ + T_CompressionAmRoutine, /* in access/compressamapi.h */ T_ForeignKeyCacheInfo, /* in utils/rel.h */ T_CallContext, /* in nodes/parsenodes.h */ T_SupportRequestSimplify, /* in nodes/supportnodes.h */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 55cab4d2bf..53d378b67d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..667927fd7c 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_info; /* Original data size (excludes header) and + * flags */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,21 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* va_info in va_compress contains raw size of datum and optional flags */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK) +#define VARFLAGS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..167878e78b --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,201 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + x | text | | | | extended | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..329e7881b0 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,189 @@ +\set HIDE_COMPRESSAM off +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+------+-----------+----------+---------+----------+--------------+------------- + f1 | text | | | | extended | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: not built with lz4 support +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: not built with lz4 support +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_COMPRESSAM on diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 7204fdb0b4..14f3fa3fc8 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -4898,8 +4898,8 @@ Indexes: -- check printing info about access methods \dA List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4907,13 +4907,15 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA * List of access methods - Name | Type ---------+------- + Name | Type +--------+------------- brin | Index btree | Index gin | Index @@ -4921,8 +4923,10 @@ List of access methods hash | Index heap | Table heap2 | Table + lz4 | Compression + pglz | Compression spgist | Index -(8 rows) +(10 rows) \dA h* List of access methods @@ -4947,32 +4951,36 @@ List of access methods \dA: extra argument "bar" ignored \dA+ - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ * - List of access methods - Name | Type | Handler | Description ---------+-------+----------------------+---------------------------------------- - brin | Index | brinhandler | block range index (BRIN) access method - btree | Index | bthandler | b-tree index access method - gin | Index | ginhandler | GIN index access method - gist | Index | gisthandler | GiST index access method - hash | Index | hashhandler | hash index access method - heap | Table | heap_tableam_handler | heap table access method - heap2 | Table | heap_tableam_handler | - spgist | Index | spghandler | SP-GiST index access method -(8 rows) + List of access methods + Name | Type | Handler | Description +--------+-------------+----------------------+---------------------------------------- + brin | Index | brinhandler | block range index (BRIN) access method + btree | Index | bthandler | b-tree index access method + gin | Index | ginhandler | GIN index access method + gist | Index | gisthandler | GiST index access method + hash | Index | hashhandler | hash index access method + heap | Table | heap_tableam_handler | heap table access method + heap2 | Table | heap_tableam_handler | + lz4 | Compression | lz4handler | lz4 compression access method + pglz | Compression | pglzhandler | pglz compression access method + spgist | Index | spghandler | SP-GiST index access method +(10 rows) \dA+ h* List of access methods diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 12bb67e491..e9c0fc9760 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 59b416fd80..4d5577bae3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -199,6 +199,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..450416ecb4 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,85 @@ +\set HIDE_COMPRESSAM off + +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_COMPRESSAM on diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index 2aa062b2c9..c64b369047 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index bab4f3adb3..46cac11d39 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionAmRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --YZ5djTAD1cGYuMQK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --C94crkcyjafcjHxo Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-3nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 1/5] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 118 +++++++ configure.ac | 18 + doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 247 ++++++++++++++ src/test/regress/expected/compression_1.out | 240 ++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 102 ++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 52 files changed, 1645 insertions(+), 85 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..761a27965d 100755 --- a/configure +++ b/configure @@ -699,6 +699,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +865,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -1569,6 +1571,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8563,6 +8566,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12148,56 @@ fi fi +if test "$with_lz4" = yes ; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13464,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes ; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..616ce5e1a2 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,10 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1420,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes ; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..031d20d5fc 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -13,6 +13,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..503d64df38 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..45e1cfa56c 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -17,6 +17,7 @@ #include <unistd.h> #include <signal.h> +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/htup_details.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9abc4a1f55..d8b71e5bc3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -29,6 +29,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/htup_details.h" #include "access/multixact.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..2196b44aae 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/toast_compression.h" #include "access/heapam.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 559fa1d2e5..d295d85cd2 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/attmap.h" +#include "access/toast_compression.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heapam_xlog.h" @@ -558,6 +559,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -852,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2396,6 +2410,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2430,6 +2460,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2675,6 +2710,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6340,6 +6388,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11859,6 +11919,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17641,3 +17718,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..9d923b5d95 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3421,11 +3423,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3710,6 +3721,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15296,6 +15308,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15816,6 +15829,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..af772ba6bc 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -27,6 +27,7 @@ #include "postgres.h" #include "access/amapi.h" +#include "access/toast_compression.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/reloptions.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..37876a8cb7 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -17,6 +17,7 @@ #include <ctype.h> #include <limits.h> +#include "access/toast_compression.h" #include "access/detoast.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index eea9f30a79..0296b9bb5e 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -160,6 +160,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTablespaces; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index eb988d7eb4..cef9bbd76a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1047,6 +1048,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8747,6 +8759,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8775,6 +8788,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15891,6 +15905,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..19d2ba26bf 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..1de3ae2646 --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,247 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..2999b3bb79 --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,240 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..ab35f3b4ae --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,102 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 select large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --nHwqXXcoX0o6fKCv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-f-use-pkgconfig.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH 4/8] Built-in compression method @ 2021-03-05 04:03 Dilip Kumar <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Dilip Kumar @ 2021-03-05 04:03 UTC (permalink / raw) Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40... https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail... https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail... --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see <http://pkg-config.freedesktop.org/;. +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attcompression</structfield> <type>char</type> + </para> + <para> + The current compression method of the column. Must be <literal>InvalidCompressionMethod</literal> + if and only if typstorage is 'plain' or 'external'. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>attacl</structfield> <type>aclitem[]</type> diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [ - { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] + { <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ] | <replaceable>table_constraint</replaceable> | LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term> + <listitem> + <para> + The <literal>COMPRESSION</literal> clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See <xref linkend="sql-altertable"/> for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are <literal>pglz</literal> and + <literal>lz4</literal>. <literal>lz4</literal> is available only if + <literal>--with-lz4</literal> was used when building + <productname>PostgreSQL</productname>. The default + is <literal>pglz</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] )</literal></term> <listitem> @@ -605,6 +625,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry> + <term><literal>INCLUDING COMPRESSION</literal></term> + <listitem> + <para> + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>INCLUDING CONSTRAINTS</literal></term> <listitem> diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar </listitem> </varlistentry> + <varlistentry> + <term><varname>HIDE_TOAST_COMPRESSION</varname></term> + <listitem> + <para> + If this variable is set to <literal>true</literal>, column + compression method details are not displayed. This is mainly + useful for regression tests. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><varname>HIDE_TABLEAM</varname></term> <listitem> diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include <lz4.h> +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <str> optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include <limits.h> #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
* [PATCH v35 1/5] Add REPACK command @ 2026-02-16 19:49 Antonin Houska <[email protected]> 0 siblings, 0 replies; 119+ messages in thread From: Antonin Houska @ 2026-02-16 19:49 UTC (permalink / raw) REPACK absorbs the functionality of VACUUM FULL and CLUSTER in a single command. Because this functionality is completely different from regular VACUUM, having it separate from VACUUM makes it easier for users to understand; as for CLUSTER, the term is heavily overloaded in the IT world and even in Postgres itself, so it's good that we can avoid it. Author: Antonin Houska <[email protected]> Co-authored-by: Álvaro Herrera <[email protected]> Reviewed-by: Mihail Nikalayeu <[email protected]> Reviewed-by: Robert Treat <[email protected]> Reviewed-by: Euler Taveira <[email protected]> Reviewed-by: Matheus Alcantara <[email protected]> Reviewed-by: Junwang Zhao <[email protected]> Reviewed-by: jian he <[email protected]> Discussion: https://postgr.es/m/82651.1720540558@antos Discussion: https://postgr.es/m/[email protected] --- doc/src/sgml/monitoring.sgml | 223 +++++- doc/src/sgml/ref/allfiles.sgml | 1 + doc/src/sgml/ref/cluster.sgml | 97 +-- doc/src/sgml/ref/repack.sgml | 327 +++++++++ doc/src/sgml/ref/vacuum.sgml | 33 +- doc/src/sgml/reference.sgml | 1 + src/backend/access/heap/heapam_handler.c | 32 +- src/backend/catalog/index.c | 2 +- src/backend/catalog/system_views.sql | 29 +- src/backend/commands/cluster.c | 848 +++++++++++++++-------- src/backend/commands/vacuum.c | 6 +- src/backend/parser/gram.y | 86 ++- src/backend/tcop/utility.c | 23 +- src/backend/utils/adt/pgstatfuncs.c | 4 +- src/bin/psql/tab-complete.in.c | 42 +- src/include/commands/cluster.h | 8 +- src/include/commands/progress.h | 48 +- src/include/nodes/parsenodes.h | 35 +- src/include/parser/kwlist.h | 1 + src/include/tcop/cmdtaglist.h | 1 + src/include/utils/backend_progress.h | 2 +- src/test/regress/expected/cluster.out | 134 +++- src/test/regress/expected/rules.out | 72 +- src/test/regress/sql/cluster.sql | 70 +- src/tools/pgindent/typedefs.list | 2 + 25 files changed, 1597 insertions(+), 530 deletions(-) create mode 100644 doc/src/sgml/ref/repack.sgml diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml index b77d189a500..71c92ed53ef 100644 --- a/doc/src/sgml/monitoring.sgml +++ b/doc/src/sgml/monitoring.sgml @@ -405,6 +405,14 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser </entry> </row> + <row> + <entry><structname>pg_stat_progress_repack</structname><indexterm><primary>pg_stat_progress_repack</primary></indexterm></entry> + <entry>One row for each backend running + <command>REPACK</command>, showing current progress. See + <xref linkend="repack-progress-reporting"/>. + </entry> + </row> + <row> <entry><structname>pg_stat_progress_basebackup</structname><indexterm><primary>pg_stat_progress_basebackup</primary></indexterm></entry> <entry>One row for each WAL sender process streaming a base backup, @@ -5646,7 +5654,8 @@ FROM pg_stat_get_backend_idset() AS backendid; certain commands during command execution. Currently, the only commands which support progress reporting are <command>ANALYZE</command>, <command>CLUSTER</command>, - <command>CREATE INDEX</command>, <command>VACUUM</command>, + <command>CREATE INDEX</command>, <command>REPACK</command>, + <command>VACUUM</command>, <command>COPY</command>, and <xref linkend="protocol-replication-base-backup"/> (i.e., replication command that <xref linkend="app-pgbasebackup"/> issues to take @@ -6130,6 +6139,218 @@ FROM pg_stat_get_backend_idset() AS backendid; </table> </sect2> + <sect2 id="repack-progress-reporting"> + <title>REPACK Progress Reporting</title> + + <indexterm> + <primary>pg_stat_progress_repack</primary> + </indexterm> + + <para> + Whenever <command>REPACK</command> is running, + the <structname>pg_stat_progress_repack</structname> view will contain a + row for each backend that is currently running the command. The tables + below describe the information that will be reported and provide + information about how to interpret it. + </para> + + <table id="pg-stat-progress-repack-view" xreflabel="pg_stat_progress_repack"> + <title><structname>pg_stat_progress_repack</structname> View</title> + <tgroup cols="1"> + <thead> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + Column Type + </para> + <para> + Description + </para></entry> + </row> + </thead> + + <tbody> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>pid</structfield> <type>integer</type> + </para> + <para> + Process ID of backend. + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>datid</structfield> <type>oid</type> + </para> + <para> + OID of the database to which this backend is connected. + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>datname</structfield> <type>name</type> + </para> + <para> + Name of the database to which this backend is connected. + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>relid</structfield> <type>oid</type> + </para> + <para> + OID of the table being repacked. + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>phase</structfield> <type>text</type> + </para> + <para> + Current processing phase. See <xref linkend="repack-phases"/>. + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>repack_index_relid</structfield> <type>oid</type> + </para> + <para> + If the table is being scanned using an index, this is the OID of the + index being used; otherwise, it is zero. + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>heap_tuples_scanned</structfield> <type>bigint</type> + </para> + <para> + Number of heap tuples scanned. + This counter only advances when the phase is + <literal>seq scanning heap</literal>, + <literal>index scanning heap</literal> + or <literal>writing new heap</literal>. + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>heap_tuples_written</structfield> <type>bigint</type> + </para> + <para> + Number of heap tuples written. + This counter only advances when the phase is + <literal>seq scanning heap</literal>, + <literal>index scanning heap</literal> + or <literal>writing new heap</literal>. + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>heap_blks_total</structfield> <type>bigint</type> + </para> + <para> + Total number of heap blocks in the table. This number is reported + as of the beginning of <literal>seq scanning heap</literal>. + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>heap_blks_scanned</structfield> <type>bigint</type> + </para> + <para> + Number of heap blocks scanned. This counter only advances when the + phase is <literal>seq scanning heap</literal>. + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>index_rebuild_count</structfield> <type>bigint</type> + </para> + <para> + Number of indexes rebuilt. This counter only advances when the phase + is <literal>rebuilding index</literal>. + </para></entry> + </row> + </tbody> + </tgroup> + </table> + + <table id="repack-phases"> + <title>REPACK Phases</title> + <tgroup cols="2"> + <colspec colname="col1" colwidth="1*"/> + <colspec colname="col2" colwidth="2*"/> + <thead> + <row> + <entry>Phase</entry> + <entry>Description</entry> + </row> + </thead> + + <tbody> + <row> + <entry><literal>initializing</literal></entry> + <entry> + The command is preparing to begin scanning the heap. This phase is + expected to be very brief. + </entry> + </row> + <row> + <entry><literal>seq scanning heap</literal></entry> + <entry> + The command is currently scanning the table using a sequential scan. + </entry> + </row> + <row> + <entry><literal>index scanning heap</literal></entry> + <entry> + <command>REPACK</command> is currently scanning the table using an index scan. + </entry> + </row> + <row> + <entry><literal>sorting tuples</literal></entry> + <entry> + <command>REPACK</command> is currently sorting tuples. + </entry> + </row> + <row> + <entry><literal>writing new heap</literal></entry> + <entry> + <command>REPACK</command> is currently writing the new heap. + </entry> + </row> + <row> + <entry><literal>swapping relation files</literal></entry> + <entry> + The command is currently swapping newly-built files into place. + </entry> + </row> + <row> + <entry><literal>rebuilding index</literal></entry> + <entry> + The command is currently rebuilding an index. + </entry> + </row> + <row> + <entry><literal>performing final cleanup</literal></entry> + <entry> + The command is performing final cleanup. When this phase is + completed, <command>REPACK</command> will end. + </entry> + </row> + </tbody> + </tgroup> + </table> + </sect2> + <sect2 id="copy-progress-reporting"> <title>COPY Progress Reporting</title> diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml index e167406c744..141ada9c50a 100644 --- a/doc/src/sgml/ref/allfiles.sgml +++ b/doc/src/sgml/ref/allfiles.sgml @@ -167,6 +167,7 @@ Complete list of usable sgml source files in this directory. <!ENTITY refreshMaterializedView SYSTEM "refresh_materialized_view.sgml"> <!ENTITY reindex SYSTEM "reindex.sgml"> <!ENTITY releaseSavepoint SYSTEM "release_savepoint.sgml"> +<!ENTITY repack SYSTEM "repack.sgml"> <!ENTITY reset SYSTEM "reset.sgml"> <!ENTITY revoke SYSTEM "revoke.sgml"> <!ENTITY rollback SYSTEM "rollback.sgml"> diff --git a/doc/src/sgml/ref/cluster.sgml b/doc/src/sgml/ref/cluster.sgml index 0b47460080b..2cda711bc9f 100644 --- a/doc/src/sgml/ref/cluster.sgml +++ b/doc/src/sgml/ref/cluster.sgml @@ -33,51 +33,13 @@ CLUSTER [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] [ <r <title>Description</title> <para> - <command>CLUSTER</command> instructs <productname>PostgreSQL</productname> - to cluster the table specified - by <replaceable class="parameter">table_name</replaceable> - based on the index specified by - <replaceable class="parameter">index_name</replaceable>. The index must - already have been defined on - <replaceable class="parameter">table_name</replaceable>. + The <command>CLUSTER</command> command is equivalent to + <xref linkend="sql-repack"/> with an <literal>USING INDEX</literal> + clause. See there for more details. </para> - <para> - When a table is clustered, it is physically reordered - based on the index information. Clustering is a one-time operation: - when the table is subsequently updated, the changes are - not clustered. That is, no attempt is made to store new or - updated rows according to their index order. (If one wishes, one can - periodically recluster by issuing the command again. Also, setting - the table's <literal>fillfactor</literal> storage parameter to less than - 100% can aid in preserving cluster ordering during updates, since updated - rows are kept on the same page if enough space is available there.) - </para> +<!-- Do we need to describe exactly which options map to what? They seem obvious to me. --> - <para> - When a table is clustered, <productname>PostgreSQL</productname> - remembers which index it was clustered by. The form - <command>CLUSTER <replaceable class="parameter">table_name</replaceable></command> - reclusters the table using the same index as before. You can also - use the <literal>CLUSTER</literal> or <literal>SET WITHOUT CLUSTER</literal> - forms of <link linkend="sql-altertable"><command>ALTER TABLE</command></link> to set the index to be used for - future cluster operations, or to clear any previous setting. - </para> - - <para> - <command>CLUSTER</command> without a - <replaceable class="parameter">table_name</replaceable> reclusters all the - previously-clustered tables in the current database that the calling user - has privileges for. This form of <command>CLUSTER</command> cannot be - executed inside a transaction block. - </para> - - <para> - When a table is being clustered, an <literal>ACCESS - EXCLUSIVE</literal> lock is acquired on it. This prevents any other - database operations (both reads and writes) from operating on the - table until the <command>CLUSTER</command> is finished. - </para> </refsect1> <refsect1> @@ -136,63 +98,12 @@ CLUSTER [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] [ <r on the table. </para> - <para> - In cases where you are accessing single rows randomly - within a table, the actual order of the data in the - table is unimportant. However, if you tend to access some - data more than others, and there is an index that groups - them together, you will benefit from using <command>CLUSTER</command>. - If you are requesting a range of indexed values from a table, or a - single indexed value that has multiple rows that match, - <command>CLUSTER</command> will help because once the index identifies the - table page for the first row that matches, all other rows - that match are probably already on the same table page, - and so you save disk accesses and speed up the query. - </para> - - <para> - <command>CLUSTER</command> can re-sort the table using either an index scan - on the specified index, or (if the index is a b-tree) a sequential - scan followed by sorting. It will attempt to choose the method that - will be faster, based on planner cost parameters and available statistical - information. - </para> - <para> While <command>CLUSTER</command> is running, the <xref linkend="guc-search-path"/> is temporarily changed to <literal>pg_catalog, pg_temp</literal>. </para> - <para> - When an index scan is used, a temporary copy of the table is created that - contains the table data in the index order. Temporary copies of each - index on the table are created as well. Therefore, you need free space on - disk at least equal to the sum of the table size and the index sizes. - </para> - - <para> - When a sequential scan and sort is used, a temporary sort file is - also created, so that the peak temporary space requirement is as much - as double the table size, plus the index sizes. This method is often - faster than the index scan method, but if the disk space requirement is - intolerable, you can disable this choice by temporarily setting <xref - linkend="guc-enable-sort"/> to <literal>off</literal>. - </para> - - <para> - It is advisable to set <xref linkend="guc-maintenance-work-mem"/> to - a reasonably large value (but not more than the amount of RAM you can - dedicate to the <command>CLUSTER</command> operation) before clustering. - </para> - - <para> - Because the planner records statistics about the ordering of - tables, it is advisable to run <link linkend="sql-analyze"><command>ANALYZE</command></link> - on the newly clustered table. - Otherwise, the planner might make poor choices of query plans. - </para> - <para> Because <command>CLUSTER</command> remembers which indexes are clustered, one can cluster the tables one wants clustered manually the first time, diff --git a/doc/src/sgml/ref/repack.sgml b/doc/src/sgml/ref/repack.sgml new file mode 100644 index 00000000000..25b55e2228b --- /dev/null +++ b/doc/src/sgml/ref/repack.sgml @@ -0,0 +1,327 @@ +<!-- +doc/src/sgml/ref/repack.sgml +PostgreSQL documentation +--> + +<refentry id="sql-repack"> + <indexterm zone="sql-repack"> + <primary>REPACK</primary> + </indexterm> + + <refmeta> + <refentrytitle>REPACK</refentrytitle> + <manvolnum>7</manvolnum> + <refmiscinfo>SQL - Language Statements</refmiscinfo> + </refmeta> + + <refnamediv> + <refname>REPACK</refname> + <refpurpose>rewrite a table to reclaim disk space</refpurpose> + </refnamediv> + + <refsynopsisdiv> +<synopsis> +REPACK [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] [ <replaceable class="parameter">table_and_columns</replaceable> [ USING INDEX [ <replaceable class="parameter">index_name</replaceable> ] ] ] +REPACK [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] USING INDEX + +<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase> + + VERBOSE [ <replaceable class="parameter">boolean</replaceable> ] + ANALYZE [ <replaceable class="parameter">boolean</replaceable> ] + +<phrase>and <replaceable class="parameter">table_and_columns</replaceable> is:</phrase> + + <replaceable class="parameter">table_name</replaceable> [ ( <replaceable class="parameter">column_name</replaceable> [, ...] ) ] +</synopsis> + </refsynopsisdiv> + + <refsect1> + <title>Description</title> + + <para> + <command>REPACK</command> reclaims storage occupied by dead + tuples. Unlike <command>VACUUM</command>, it does so by rewriting the + entire contents of the table specified + by <replaceable class="parameter">table_name</replaceable> into a new disk + file with no extra space (except for the space guaranteed by + the <literal>fillfactor</literal> storage parameter), allowing unused space + to be returned to the operating system. + </para> + + <para> + Without + a <replaceable class="parameter">table_name</replaceable>, <command>REPACK</command> + processes every table and materialized view in the current database that + the current user has the <literal>MAINTAIN</literal> privilege on. This + form of <command>REPACK</command> cannot be executed inside a transaction + block. + </para> + + <para> + If a <literal>USING INDEX</literal> clause is specified, the rows are + physically reordered based on information from an index. Please see the + notes on clustering below. + </para> + + <para> + When a table is being repacked, an <literal>ACCESS EXCLUSIVE</literal> lock + is acquired on it. This prevents any other database operations (both reads + and writes) from operating on the table until the <command>REPACK</command> + is finished. + </para> + + <refsect2 id="sql-repack-notes-on-clustering" xreflabel="Notes on Clustering"> + <title>Notes on Clustering</title> + + <para> + If the <literal>USING INDEX</literal> clause is specified, the rows in + the table are physically reordered following an index: if an index name + is specified in the command, then that index is used; if no index name + is specified, then the index that has been configured as the index to + cluster on. If no index has been configured in this way, an error is + thrown. The index given in the <literal>USING INDEX</literal> clause + is configured as the index to cluster on, as well as an index given + to the <command>CLUSTER</command> command. An index can be set + manually using <command>ALTER TABLE ... CLUSTER ON</command>, and reset + with <command>ALTER TABLE ... SET WITHOUT CLUSTER</command>. + </para> + + <para> + If no table name is specified in <command>REPACK USING INDEX</command>, + all tables which have a clustering index defined and which the calling + user has privileges for are processed. + </para> + + <para> + Clustering is a one-time operation: when the table is + subsequently updated, the changes are not clustered. That is, no attempt + is made to store new or updated rows according to their index order. (If + one wishes, one can periodically recluster by issuing the command again. + Also, setting the table's <literal>fillfactor</literal> storage parameter + to less than 100% can aid in preserving cluster ordering during updates, + since updated rows are kept on the same page if enough space is available + there.) + </para> + + <para> + In cases where you are accessing single rows randomly within a table, the + actual order of the data in the table is unimportant. However, if you tend + to access some data more than others, and there is an index that groups + them together, you will benefit from using clustering. If + you are requesting a range of indexed values from a table, or a single + indexed value that has multiple rows that match, + <command>REPACK</command> will help because once the index identifies the + table page for the first row that matches, all other rows that match are + probably already on the same table page, and so you save disk accesses and + speed up the query. + </para> + + <para> + <command>REPACK</command> can re-sort the table using either an index scan + on the specified index (if the index is a b-tree), or a sequential scan + followed by sorting. It will attempt to choose the method that will be + faster, based on planner cost parameters and available statistical + information. + </para> + + <para> + Because the planner records statistics about the ordering of tables, it is + advisable to + run <link linkend="sql-analyze"><command>ANALYZE</command></link> on the + newly repacked table. Otherwise, the planner might make poor choices of + query plans. + </para> + </refsect2> + + <refsect2 id="sql-repack-notes-on-resources" xreflabel="Notes on Resources"> + <title>Notes on Resources</title> + + <para> + When an index scan or a sequential scan without sort is used, a temporary + copy of the table is created that contains the table data in the index + order. Temporary copies of each index on the table are created as well. + Therefore, you need free space on disk at least equal to the sum of the + table size and the index sizes. + </para> + + <para> + When a sequential scan and sort is used, a temporary sort file is also + created, so that the peak temporary space requirement is as much as double + the table size, plus the index sizes. This method is often faster than + the index scan method, but if the disk space requirement is intolerable, + you can disable this choice by temporarily setting + <xref linkend="guc-enable-sort"/> to <literal>off</literal>. + </para> + + <para> + It is advisable to set <xref linkend="guc-maintenance-work-mem"/> to a + reasonably large value (but not more than the amount of RAM you can + dedicate to the <command>REPACK</command> operation) before repacking. + </para> + </refsect2> + + </refsect1> + + <refsect1> + <title>Parameters</title> + + <variablelist> + <varlistentry> + <term><replaceable class="parameter">table_name</replaceable></term> + <listitem> + <para> + The name (possibly schema-qualified) of a table. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">column_name</replaceable></term> + <listitem> + <para> + The name of a specific column to analyze. Defaults to all columns. + If a column list is specific, <literal>ANALYZE</literal> must also + be specified. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">index_name</replaceable></term> + <listitem> + <para> + The name of an index. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>VERBOSE</literal></term> + <listitem> + <para> + Prints a progress report as each table is repacked + at <literal>INFO</literal> level. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>ANALYZE</literal></term> + <term><literal>ANALYSE</literal></term> + <listitem> + <para> + Applies <xref linkend="sql-analyze"/> on the table after repacking. This is + currently only supported when a single (non-partitioned) table is specified. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">boolean</replaceable></term> + <listitem> + <para> + Specifies whether the selected option should be turned on or off. + You can write <literal>TRUE</literal>, <literal>ON</literal>, or + <literal>1</literal> to enable the option, and <literal>FALSE</literal>, + <literal>OFF</literal>, or <literal>0</literal> to disable it. The + <replaceable class="parameter">boolean</replaceable> value can also + be omitted, in which case <literal>TRUE</literal> is assumed. + </para> + </listitem> + </varlistentry> + </variablelist> + </refsect1> + + <refsect1> + <title>Notes</title> + + <para> + To repack a table, one must have the <literal>MAINTAIN</literal> privilege + on the table. + </para> + + <para> + While <command>REPACK</command> is running, the <xref + linkend="guc-search-path"/> is temporarily changed to <literal>pg_catalog, + pg_temp</literal>. + </para> + + <para> + Each backend running <command>REPACK</command> will report its progress + in the <structname>pg_stat_progress_repack</structname> view. See + <xref linkend="repack-progress-reporting"/> for details. + </para> + + <para> + Repacking a partitioned table repacks each of its partitions. If an index + is specified, each partition is repacked using the partition of that + index. <command>REPACK</command> on a partitioned table cannot be executed + inside a transaction block. + </para> + + </refsect1> + + <refsect1> + <title>Examples</title> + + <para> + Repack the table <literal>employees</literal>: +<programlisting> +REPACK employees; +</programlisting> + </para> + + <para> + Repack the table <literal>employees</literal> on the basis of its + index <literal>employees_ind</literal> (Since index is used here, this is + effectively clustering): +<programlisting> +REPACK employees USING INDEX employees_ind; +</programlisting> + </para> + + <para> + Repack the table <literal>cases</literal> on physical ordering, + running an <command>ANALYZE</command> on the given columns once + repacking is done, showing informational messages: +<programlisting> +REPACK (ANALYZE, VERBOSE) cases (district, case_nr); +</programlisting> + </para> + + <para> + Repack all tables in the database on which you have + the <literal>MAINTAIN</literal> privilege: +<programlisting> +REPACK; +</programlisting> + </para> + + <para> + Repack all tables for which a clustering index has previously been + configured on which you have the <literal>MAINTAIN</literal> privilege, + showing informational messages: +<programlisting> +REPACK (VERBOSE) USING INDEX; +</programlisting> + </para> + + </refsect1> + + <refsect1> + <title>Compatibility</title> + + <para> + There is no <command>REPACK</command> statement in the SQL standard. + </para> + </refsect1> + + <refsect1> + <title>See Also</title> + + <simplelist type="inline"> + <member><xref linkend="repack-progress-reporting"/></member> + </simplelist> + </refsect1> + +</refentry> diff --git a/doc/src/sgml/ref/vacuum.sgml b/doc/src/sgml/ref/vacuum.sgml index 6d0fdd43cfb..ac5d083d468 100644 --- a/doc/src/sgml/ref/vacuum.sgml +++ b/doc/src/sgml/ref/vacuum.sgml @@ -25,7 +25,6 @@ VACUUM [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] [ <re <phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase> - FULL [ <replaceable class="parameter">boolean</replaceable> ] FREEZE [ <replaceable class="parameter">boolean</replaceable> ] VERBOSE [ <replaceable class="parameter">boolean</replaceable> ] ANALYZE [ <replaceable class="parameter">boolean</replaceable> ] @@ -39,6 +38,7 @@ VACUUM [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] [ <re SKIP_DATABASE_STATS [ <replaceable class="parameter">boolean</replaceable> ] ONLY_DATABASE_STATS [ <replaceable class="parameter">boolean</replaceable> ] BUFFER_USAGE_LIMIT <replaceable class="parameter">size</replaceable> + FULL [ <replaceable class="parameter">boolean</replaceable> ] <phrase>and <replaceable class="parameter">table_and_columns</replaceable> is:</phrase> @@ -95,20 +95,6 @@ VACUUM [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] [ <re <title>Parameters</title> <variablelist> - <varlistentry> - <term><literal>FULL</literal></term> - <listitem> - <para> - Selects <quote>full</quote> vacuum, which can reclaim more - space, but takes much longer and exclusively locks the table. - This method also requires extra disk space, since it writes a - new copy of the table and doesn't release the old copy until - the operation is complete. Usually this should only be used when a - significant amount of space needs to be reclaimed from within the table. - </para> - </listitem> - </varlistentry> - <varlistentry> <term><literal>FREEZE</literal></term> <listitem> @@ -362,6 +348,23 @@ VACUUM [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] [ <re </listitem> </varlistentry> + <varlistentry> + <term><literal>FULL</literal></term> + <listitem> + <para> + This option, which is deprecated, makes <command>VACUUM</command> + behave like <command>REPACK</command> without a + <literal>USING INDEX</literal> clause. + This method of compacting the table takes much longer than + <command>VACUUM</command> and exclusively locks the table. + This method also requires extra disk space, since it writes a + new copy of the table and doesn't release the old copy until + the operation is complete. Usually this should only be used when a + significant amount of space needs to be reclaimed from within the table. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><replaceable class="parameter">boolean</replaceable></term> <listitem> diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml index 2cf02c37b17..d9fdbb5d254 100644 --- a/doc/src/sgml/reference.sgml +++ b/doc/src/sgml/reference.sgml @@ -195,6 +195,7 @@ &refreshMaterializedView; &reindex; &releaseSavepoint; + &repack; &reset; &revoke; &rollback; diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index b83e2013d50..e3748ff6301 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -741,13 +741,13 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (OldIndex != NULL && !use_sort) { const int ci_index[] = { - PROGRESS_CLUSTER_PHASE, - PROGRESS_CLUSTER_INDEX_RELID + PROGRESS_REPACK_PHASE, + PROGRESS_REPACK_INDEX_RELID }; int64 ci_val[2]; /* Set phase and OIDOldIndex to columns */ - ci_val[0] = PROGRESS_CLUSTER_PHASE_INDEX_SCAN_HEAP; + ci_val[0] = PROGRESS_REPACK_PHASE_INDEX_SCAN_HEAP; ci_val[1] = RelationGetRelid(OldIndex); pgstat_progress_update_multi_param(2, ci_index, ci_val); @@ -759,15 +759,15 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, else { /* In scan-and-sort mode and also VACUUM FULL, set phase */ - pgstat_progress_update_param(PROGRESS_CLUSTER_PHASE, - PROGRESS_CLUSTER_PHASE_SEQ_SCAN_HEAP); + pgstat_progress_update_param(PROGRESS_REPACK_PHASE, + PROGRESS_REPACK_PHASE_SEQ_SCAN_HEAP); tableScan = table_beginscan(OldHeap, SnapshotAny, 0, (ScanKey) NULL); heapScan = (HeapScanDesc) tableScan; indexScan = NULL; /* Set total heap blocks */ - pgstat_progress_update_param(PROGRESS_CLUSTER_TOTAL_HEAP_BLKS, + pgstat_progress_update_param(PROGRESS_REPACK_TOTAL_HEAP_BLKS, heapScan->rs_nblocks); } @@ -809,7 +809,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, * is manually updated to the correct value when the table * scan finishes. */ - pgstat_progress_update_param(PROGRESS_CLUSTER_HEAP_BLKS_SCANNED, + pgstat_progress_update_param(PROGRESS_REPACK_HEAP_BLKS_SCANNED, heapScan->rs_nblocks); break; } @@ -825,7 +825,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ if (prev_cblock != heapScan->rs_cblock) { - pgstat_progress_update_param(PROGRESS_CLUSTER_HEAP_BLKS_SCANNED, + pgstat_progress_update_param(PROGRESS_REPACK_HEAP_BLKS_SCANNED, (heapScan->rs_cblock + heapScan->rs_nblocks - heapScan->rs_startblock @@ -926,14 +926,14 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, * In scan-and-sort mode, report increase in number of tuples * scanned */ - pgstat_progress_update_param(PROGRESS_CLUSTER_HEAP_TUPLES_SCANNED, + pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_SCANNED, *num_tuples); } else { const int ct_index[] = { - PROGRESS_CLUSTER_HEAP_TUPLES_SCANNED, - PROGRESS_CLUSTER_HEAP_TUPLES_WRITTEN + PROGRESS_REPACK_HEAP_TUPLES_SCANNED, + PROGRESS_REPACK_HEAP_TUPLES_WRITTEN }; int64 ct_val[2]; @@ -966,14 +966,14 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, double n_tuples = 0; /* Report that we are now sorting tuples */ - pgstat_progress_update_param(PROGRESS_CLUSTER_PHASE, - PROGRESS_CLUSTER_PHASE_SORT_TUPLES); + pgstat_progress_update_param(PROGRESS_REPACK_PHASE, + PROGRESS_REPACK_PHASE_SORT_TUPLES); tuplesort_performsort(tuplesort); /* Report that we are now writing new heap */ - pgstat_progress_update_param(PROGRESS_CLUSTER_PHASE, - PROGRESS_CLUSTER_PHASE_WRITE_NEW_HEAP); + pgstat_progress_update_param(PROGRESS_REPACK_PHASE, + PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); for (;;) { @@ -991,7 +991,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, values, isnull, rwstate); /* Report n_tuples */ - pgstat_progress_update_param(PROGRESS_CLUSTER_HEAP_TUPLES_WRITTEN, + pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_WRITTEN, n_tuples); } diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 43de42ce39e..5ee6389d39c 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -4077,7 +4077,7 @@ reindex_relation(const ReindexStmt *stmt, Oid relid, int flags, Assert(!ReindexIsProcessingIndex(indexOid)); /* Set index rebuild count */ - pgstat_progress_update_param(PROGRESS_CLUSTER_INDEX_REBUILD_COUNT, + pgstat_progress_update_param(PROGRESS_REPACK_INDEX_REBUILD_COUNT, i); i++; } diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index 1ea8f1faa9e..201c503548e 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1304,14 +1304,15 @@ CREATE VIEW pg_stat_progress_vacuum AS FROM pg_stat_get_progress_info('VACUUM') AS S LEFT JOIN pg_database D ON S.datid = D.oid; -CREATE VIEW pg_stat_progress_cluster AS +CREATE VIEW pg_stat_progress_repack AS SELECT S.pid AS pid, S.datid AS datid, D.datname AS datname, S.relid AS relid, CASE S.param1 WHEN 1 THEN 'CLUSTER' - WHEN 2 THEN 'VACUUM FULL' + WHEN 2 THEN 'REPACK' + WHEN 3 THEN 'VACUUM FULL' END AS command, CASE S.param2 WHEN 0 THEN 'initializing' WHEN 1 THEN 'seq scanning heap' @@ -1322,15 +1323,35 @@ CREATE VIEW pg_stat_progress_cluster AS WHEN 6 THEN 'rebuilding index' WHEN 7 THEN 'performing final cleanup' END AS phase, - CAST(S.param3 AS oid) AS cluster_index_relid, + CAST(S.param3 AS oid) AS repack_index_relid, S.param4 AS heap_tuples_scanned, S.param5 AS heap_tuples_written, S.param6 AS heap_blks_total, S.param7 AS heap_blks_scanned, S.param8 AS index_rebuild_count - FROM pg_stat_get_progress_info('CLUSTER') AS S + FROM pg_stat_get_progress_info('REPACK') AS S LEFT JOIN pg_database D ON S.datid = D.oid; +-- This view is as the one above, except for renaming a column and avoiding +-- 'REPACK' as a command name to report. +CREATE VIEW pg_stat_progress_cluster AS + SELECT + pid, + datid, + datname, + relid, + CASE WHEN command IN ('CLUSTER', 'VACUUM FULL') THEN command + WHEN repack_index_relid = 0 THEN 'VACUUM FULL' + ELSE 'CLUSTER' END AS command, + phase, + repack_index_relid AS cluster_index_relid, + heap_tuples_scanned, + heap_tuples_written, + heap_blks_total, + heap_blks_scanned, + index_rebuild_count + FROM pg_stat_progress_repack; + CREATE VIEW pg_stat_progress_create_index AS SELECT S.pid AS pid, S.datid AS datid, D.datname AS datname, diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 60a4617a585..d70636a6a63 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1,7 +1,8 @@ /*------------------------------------------------------------------------- * * cluster.c - * CLUSTER a table on an index. This is now also used for VACUUM FULL. + * CLUSTER a table on an index. This is now also used for VACUUM FULL and + * REPACK. * * There is hardly anything left of Paul Brown's original implementation... * @@ -67,27 +68,35 @@ typedef struct Oid indexOid; } RelToCluster; - -static void cluster_multiple_rels(List *rtcs, ClusterParams *params); +static bool cluster_rel_recheck(RepackCommand cmd, Relation OldHeap, + Oid indexOid, Oid userid, int options); static void rebuild_relation(Relation OldHeap, Relation index, bool verbose); static void copy_table_data(Relation NewHeap, Relation OldHeap, Relation OldIndex, bool verbose, bool *pSwapToastByContent, TransactionId *pFreezeXid, MultiXactId *pCutoffMulti); -static List *get_tables_to_cluster(MemoryContext cluster_context); -static List *get_tables_to_cluster_partitioned(MemoryContext cluster_context, - Oid indexOid); -static bool cluster_is_permitted_for_relation(Oid relid, Oid userid); +static List *get_tables_to_repack(RepackCommand cmd, bool usingindex, + MemoryContext permcxt); +static List *get_tables_to_repack_partitioned(RepackCommand cmd, + Oid relid, bool rel_is_index, + MemoryContext permcxt); +static bool cluster_is_permitted_for_relation(RepackCommand cmd, + Oid relid, Oid userid); +static Relation process_single_relation(RepackStmt *stmt, + ClusterParams *params); +static Oid determine_clustered_index(Relation rel, bool usingindex, + const char *indexname); +static const char *RepackCommandAsString(RepackCommand cmd); -/*--------------------------------------------------------------------------- - * This cluster code allows for clustering multiple tables at once. Because +/* + * The repack code allows for processing multiple tables at once. Because * of this, we cannot just run everything on a single transaction, or we * would be forced to acquire exclusive locks on all the tables being * clustered, simultaneously --- very likely leading to deadlock. * - * To solve this we follow a similar strategy to VACUUM code, - * clustering each relation in a separate transaction. For this to work, - * we need to: + * To solve this we follow a similar strategy to VACUUM code, processing each + * relation in a separate transaction. For this to work, we need to: + * * - provide a separate memory context so that we can pass information in * a way that survives across transactions * - start a new transaction every time a new relation is clustered @@ -98,197 +107,166 @@ static bool cluster_is_permitted_for_relation(Oid relid, Oid userid); * * The single-relation case does not have any such overhead. * - * We also allow a relation to be specified without index. In that case, - * the indisclustered bit will be looked up, and an ERROR will be thrown - * if there is no index with the bit set. - *--------------------------------------------------------------------------- + * We also allow a relation to be repacked following an index, but without + * naming a specific one. In that case, the indisclustered bit will be + * looked up, and an ERROR will be thrown if no so-marked index is found. */ void -cluster(ParseState *pstate, ClusterStmt *stmt, bool isTopLevel) +ExecRepack(ParseState *pstate, RepackStmt *stmt, bool isTopLevel) { - ListCell *lc; ClusterParams params = {0}; - bool verbose = false; Relation rel = NULL; - Oid indexOid = InvalidOid; - MemoryContext cluster_context; + MemoryContext repack_context; List *rtcs; /* Parse option list */ - foreach(lc, stmt->params) + foreach_node(DefElem, opt, stmt->params) { - DefElem *opt = (DefElem *) lfirst(lc); - if (strcmp(opt->defname, "verbose") == 0) - verbose = defGetBoolean(opt); + params.options |= defGetBoolean(opt) ? CLUOPT_VERBOSE : 0; + else if (strcmp(opt->defname, "analyze") == 0 || + strcmp(opt->defname, "analyse") == 0) + params.options |= defGetBoolean(opt) ? CLUOPT_ANALYZE : 0; else ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - errmsg("unrecognized %s option \"%s\"", - "CLUSTER", opt->defname), - parser_errposition(pstate, opt->location))); + errcode(ERRCODE_SYNTAX_ERROR), + errmsg("unrecognized %s option \"%s\"", + RepackCommandAsString(stmt->command), + opt->defname), + parser_errposition(pstate, opt->location)); } - params.options = (verbose ? CLUOPT_VERBOSE : 0); - + /* + * If a single relation is specified, process it and we're done ... unless + * the relation is a partitioned table, in which case we fall through. + */ if (stmt->relation != NULL) { - /* This is the single-relation case. */ - Oid tableOid; - - /* - * Find, lock, and check permissions on the table. We obtain - * AccessExclusiveLock right away to avoid lock-upgrade hazard in the - * single-transaction case. - */ - tableOid = RangeVarGetRelidExtended(stmt->relation, - AccessExclusiveLock, - 0, - RangeVarCallbackMaintainsTable, - NULL); - rel = table_open(tableOid, NoLock); - - /* - * Reject clustering a remote temp table ... their local buffer - * manager is not going to cope. - */ - if (RELATION_IS_OTHER_TEMP(rel)) - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("cannot cluster temporary tables of other sessions"))); - - if (stmt->indexname == NULL) - { - ListCell *index; - - /* We need to find the index that has indisclustered set. */ - foreach(index, RelationGetIndexList(rel)) - { - indexOid = lfirst_oid(index); - if (get_index_isclustered(indexOid)) - break; - indexOid = InvalidOid; - } - - if (!OidIsValid(indexOid)) - ereport(ERROR, - (errcode(ERRCODE_UNDEFINED_OBJECT), - errmsg("there is no previously clustered index for table \"%s\"", - stmt->relation->relname))); - } - else - { - /* - * The index is expected to be in the same namespace as the - * relation. - */ - indexOid = get_relname_relid(stmt->indexname, - rel->rd_rel->relnamespace); - if (!OidIsValid(indexOid)) - ereport(ERROR, - (errcode(ERRCODE_UNDEFINED_OBJECT), - errmsg("index \"%s\" for table \"%s\" does not exist", - stmt->indexname, stmt->relation->relname))); - } - - /* For non-partitioned tables, do what we came here to do. */ - if (rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE) - { - cluster_rel(rel, indexOid, ¶ms); - /* cluster_rel closes the relation, but keeps lock */ - - return; - } + rel = process_single_relation(stmt, ¶ms); + if (rel == NULL) + return; /* all done */ } + /* + * Don't allow ANALYZE in the multiple-relation case for now. Maybe we + * can add support for this later. + */ + if (params.options & CLUOPT_ANALYZE) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot %s multiple tables", "REPACK (ANALYZE)")); + /* * By here, we know we are in a multi-table situation. In order to avoid * holding locks for too long, we want to process each table in its own * transaction. This forces us to disallow running inside a user * transaction block. */ - PreventInTransactionBlock(isTopLevel, "CLUSTER"); + PreventInTransactionBlock(isTopLevel, RepackCommandAsString(stmt->command)); /* Also, we need a memory context to hold our list of relations */ - cluster_context = AllocSetContextCreate(PortalContext, - "Cluster", - ALLOCSET_DEFAULT_SIZES); + repack_context = AllocSetContextCreate(PortalContext, + "Repack", + ALLOCSET_DEFAULT_SIZES); + + params.options |= CLUOPT_RECHECK; /* - * Either we're processing a partitioned table, or we were not given any - * table name at all. In either case, obtain a list of relations to - * process. - * - * In the former case, an index name must have been given, so we don't - * need to recheck its "indisclustered" bit, but we have to check that it - * is an index that we can cluster on. In the latter case, we set the - * option bit to have indisclustered verified. - * - * Rechecking the relation itself is necessary here in all cases. + * If we don't have a relation yet, determine a relation list. If we do, + * then it must be a partitioned table, and we want to process its + * partitions. */ - params.options |= CLUOPT_RECHECK; - if (rel != NULL) + if (rel == NULL) { - Assert(rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE); - check_index_is_clusterable(rel, indexOid, AccessShareLock); - rtcs = get_tables_to_cluster_partitioned(cluster_context, indexOid); - - /* close relation, releasing lock on parent table */ - table_close(rel, AccessExclusiveLock); + Assert(stmt->indexname == NULL); + rtcs = get_tables_to_repack(stmt->command, stmt->usingindex, + repack_context); + params.options |= CLUOPT_RECHECK_ISCLUSTERED; } else { - rtcs = get_tables_to_cluster(cluster_context); - params.options |= CLUOPT_RECHECK_ISCLUSTERED; + Oid relid; + bool rel_is_index; + + Assert(rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE); + + /* + * If USING INDEX was specified, resolve the index name now and pass + * it down. + */ + if (stmt->usingindex) + { + /* + * If no index name was specified when repacking a partitioned + * table, punt for now. Maybe we can improve this later. + */ + if (!stmt->indexname) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("there is no previously clustered index for table \"%s\"", + RelationGetRelationName(rel))); + + relid = determine_clustered_index(rel, stmt->usingindex, + stmt->indexname); + if (!OidIsValid(relid)) + elog(ERROR, "unable to determine index to cluster on"); + /* XXX is this the right place for this check? */ + check_index_is_clusterable(rel, relid, AccessExclusiveLock); + rel_is_index = true; + } + else + { + relid = RelationGetRelid(rel); + rel_is_index = false; + } + + rtcs = get_tables_to_repack_partitioned(stmt->command, + relid, rel_is_index, + repack_context); + + /* close parent relation, releasing lock on it */ + table_close(rel, AccessExclusiveLock); + rel = NULL; } - /* Do the job. */ - cluster_multiple_rels(rtcs, ¶ms); - - /* Start a new transaction for the cleanup work. */ - StartTransactionCommand(); - - /* Clean up working storage */ - MemoryContextDelete(cluster_context); -} - -/* - * Given a list of relations to cluster, process each of them in a separate - * transaction. - * - * We expect to be in a transaction at start, but there isn't one when we - * return. - */ -static void -cluster_multiple_rels(List *rtcs, ClusterParams *params) -{ - ListCell *lc; - /* Commit to get out of starting transaction */ PopActiveSnapshot(); CommitTransactionCommand(); /* Cluster the tables, each in a separate transaction */ - foreach(lc, rtcs) + Assert(rel == NULL); + foreach_ptr(RelToCluster, rtc, rtcs) { - RelToCluster *rtc = (RelToCluster *) lfirst(lc); - Relation rel; - /* Start a new transaction for each relation. */ StartTransactionCommand(); + /* + * Open the target table, coping with the case where it has been + * dropped. + */ + rel = try_table_open(rtc->tableOid, AccessExclusiveLock); + if (rel == NULL) + { + CommitTransactionCommand(); + continue; + } + /* functions in indexes may want a snapshot set */ PushActiveSnapshot(GetTransactionSnapshot()); - rel = table_open(rtc->tableOid, AccessExclusiveLock); - /* Process this table */ - cluster_rel(rel, rtc->indexOid, params); + cluster_rel(stmt->command, rel, rtc->indexOid, ¶ms); /* cluster_rel closes the relation, but keeps lock */ PopActiveSnapshot(); CommitTransactionCommand(); } + + /* Start a new transaction for the cleanup work. */ + StartTransactionCommand(); + + /* Clean up working storage */ + MemoryContextDelete(repack_context); } /* @@ -304,11 +282,14 @@ cluster_multiple_rels(List *rtcs, ClusterParams *params) * them incrementally while we load the table. * * If indexOid is InvalidOid, the table will be rewritten in physical order - * instead of index order. This is the new implementation of VACUUM FULL, - * and error messages should refer to the operation as VACUUM not CLUSTER. + * instead of index order. + * + * 'cmd' indicates which command is being executed, to be used for error + * messages. */ void -cluster_rel(Relation OldHeap, Oid indexOid, ClusterParams *params) +cluster_rel(RepackCommand cmd, Relation OldHeap, Oid indexOid, + ClusterParams *params) { Oid tableOid = RelationGetRelid(OldHeap); Oid save_userid; @@ -323,13 +304,8 @@ cluster_rel(Relation OldHeap, Oid indexOid, ClusterParams *params) /* Check for user-requested abort. */ CHECK_FOR_INTERRUPTS(); - pgstat_progress_start_command(PROGRESS_COMMAND_CLUSTER, tableOid); - if (OidIsValid(indexOid)) - pgstat_progress_update_param(PROGRESS_CLUSTER_COMMAND, - PROGRESS_CLUSTER_COMMAND_CLUSTER); - else - pgstat_progress_update_param(PROGRESS_CLUSTER_COMMAND, - PROGRESS_CLUSTER_COMMAND_VACUUM_FULL); + pgstat_progress_start_command(PROGRESS_COMMAND_REPACK, tableOid); + pgstat_progress_update_param(PROGRESS_REPACK_COMMAND, cmd); /* * Switch to the table owner's userid, so that any index functions are run @@ -350,86 +326,38 @@ cluster_rel(Relation OldHeap, Oid indexOid, ClusterParams *params) * *must* skip the one on indisclustered since it would reject an attempt * to cluster a not-previously-clustered index. */ - if (recheck) - { - /* Check that the user still has privileges for the relation */ - if (!cluster_is_permitted_for_relation(tableOid, save_userid)) - { - relation_close(OldHeap, AccessExclusiveLock); - goto out; - } - - /* - * Silently skip a temp table for a remote session. Only doing this - * check in the "recheck" case is appropriate (which currently means - * somebody is executing a database-wide CLUSTER or on a partitioned - * table), because there is another check in cluster() which will stop - * any attempt to cluster remote temp tables by name. There is - * another check in cluster_rel which is redundant, but we leave it - * for extra safety. - */ - if (RELATION_IS_OTHER_TEMP(OldHeap)) - { - relation_close(OldHeap, AccessExclusiveLock); - goto out; - } - - if (OidIsValid(indexOid)) - { - /* - * Check that the index still exists - */ - if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(indexOid))) - { - relation_close(OldHeap, AccessExclusiveLock); - goto out; - } - - /* - * Check that the index is still the one with indisclustered set, - * if needed. - */ - if ((params->options & CLUOPT_RECHECK_ISCLUSTERED) != 0 && - !get_index_isclustered(indexOid)) - { - relation_close(OldHeap, AccessExclusiveLock); - goto out; - } - } - } + if (recheck && + !cluster_rel_recheck(cmd, OldHeap, indexOid, save_userid, + params->options)) + goto out; /* - * We allow VACUUM FULL, but not CLUSTER, on shared catalogs. CLUSTER - * would work in most respects, but the index would only get marked as - * indisclustered in the current database, leading to unexpected behavior - * if CLUSTER were later invoked in another database. + * We allow repacking shared catalogs only when not using an index. It + * would work to use an index in most respects, but the index would only + * get marked as indisclustered in the current database, leading to + * unexpected behavior if CLUSTER were later invoked in another database. */ if (OidIsValid(indexOid) && OldHeap->rd_rel->relisshared) ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("cannot cluster a shared catalog"))); + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot run %s on a shared catalog", + RepackCommandAsString(cmd))); /* * Don't process temp tables of other backends ... their local buffer * manager is not going to cope. */ if (RELATION_IS_OTHER_TEMP(OldHeap)) - { - if (OidIsValid(indexOid)) - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("cannot cluster temporary tables of other sessions"))); - else - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("cannot vacuum temporary tables of other sessions"))); - } + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot run %s on temporary tables of other sessions", + RepackCommandAsString(cmd))); /* * Also check for active uses of the relation in the current transaction, * including open scans and pending AFTER trigger events. */ - CheckTableNotInUse(OldHeap, OidIsValid(indexOid) ? "CLUSTER" : "VACUUM"); + CheckTableNotInUse(OldHeap, RepackCommandAsString(cmd)); /* Check heap and index are valid to cluster on */ if (OidIsValid(indexOid)) @@ -442,6 +370,24 @@ cluster_rel(Relation OldHeap, Oid indexOid, ClusterParams *params) else index = NULL; + /* + * When allow_system_table_mods is turned off, we disallow repacking a + * catalog on a particular index unless that's already the clustered index + * for that catalog. + * + * XXX We don't check for this in CLUSTER, because it's historically been + * allowed. + */ + if (cmd != REPACK_COMMAND_CLUSTER && + !allowSystemTableMods && OidIsValid(indexOid) && + IsCatalogRelation(OldHeap) && !index->rd_index->indisclustered) + ereport(ERROR, + errcode(ERRCODE_INSUFFICIENT_PRIVILEGE), + errmsg("permission denied: \"%s\" is a system catalog", + RelationGetRelationName(OldHeap)), + errdetail("System catalogs can only be clustered by the index they're already clustered on, if any, unless \"%s\" is enabled.", + "allow_system_table_mods")); + /* * Quietly ignore the request if this is a materialized view which has not * been populated from its query. No harm is done because there is no data @@ -482,6 +428,63 @@ out: pgstat_progress_end_command(); } +/* + * Check if the table (and its index) still meets the requirements of + * cluster_rel(). + */ +static bool +cluster_rel_recheck(RepackCommand cmd, Relation OldHeap, Oid indexOid, + Oid userid, int options) +{ + Oid tableOid = RelationGetRelid(OldHeap); + + /* Check that the user still has privileges for the relation */ + if (!cluster_is_permitted_for_relation(cmd, tableOid, userid)) + { + relation_close(OldHeap, AccessExclusiveLock); + return false; + } + + /* + * Silently skip a temp table for a remote session. Only doing this check + * in the "recheck" case is appropriate (which currently means somebody is + * executing a database-wide CLUSTER or on a partitioned table), because + * there is another check in cluster() which will stop any attempt to + * cluster remote temp tables by name. There is another check in + * cluster_rel which is redundant, but we leave it for extra safety. + */ + if (RELATION_IS_OTHER_TEMP(OldHeap)) + { + relation_close(OldHeap, AccessExclusiveLock); + return false; + } + + if (OidIsValid(indexOid)) + { + /* + * Check that the index still exists + */ + if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(indexOid))) + { + relation_close(OldHeap, AccessExclusiveLock); + return false; + } + + /* + * Check that the index is still the one with indisclustered set, if + * needed. + */ + if ((options & CLUOPT_RECHECK_ISCLUSTERED) != 0 && + !get_index_isclustered(indexOid)) + { + relation_close(OldHeap, AccessExclusiveLock); + return false; + } + } + + return true; +} + /* * Verify that the specified heap and index are valid to cluster on * @@ -642,8 +645,8 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose) Assert(CheckRelationLockedByMe(OldHeap, AccessExclusiveLock, false) && (index == NULL || CheckRelationLockedByMe(index, AccessExclusiveLock, false))); - if (index) - /* Mark the correct index as clustered */ + /* for CLUSTER or REPACK USING INDEX, mark the index as the one to use */ + if (index != NULL) mark_index_clustered(OldHeap, RelationGetRelid(index), true); /* Remember info about rel before closing OldHeap */ @@ -958,20 +961,20 @@ copy_table_data(Relation NewHeap, Relation OldHeap, Relation OldIndex, bool verb /* Log what we're doing */ if (OldIndex != NULL && !use_sort) ereport(elevel, - (errmsg("clustering \"%s.%s\" using index scan on \"%s\"", - nspname, - RelationGetRelationName(OldHeap), - RelationGetRelationName(OldIndex)))); + errmsg("repacking \"%s.%s\" using index scan on \"%s\"", + nspname, + RelationGetRelationName(OldHeap), + RelationGetRelationName(OldIndex))); else if (use_sort) ereport(elevel, - (errmsg("clustering \"%s.%s\" using sequential scan and sort", - nspname, - RelationGetRelationName(OldHeap)))); + errmsg("repacking \"%s.%s\" using sequential scan and sort", + nspname, + RelationGetRelationName(OldHeap))); else ereport(elevel, - (errmsg("vacuuming \"%s.%s\"", - nspname, - RelationGetRelationName(OldHeap)))); + errmsg("repacking \"%s.%s\" in physical order", + nspname, + RelationGetRelationName(OldHeap))); /* * Hand off the actual copying to AM specific function, the generic code @@ -1458,8 +1461,8 @@ finish_heap_swap(Oid OIDOldHeap, Oid OIDNewHeap, int i; /* Report that we are now swapping relation files */ - pgstat_progress_update_param(PROGRESS_CLUSTER_PHASE, - PROGRESS_CLUSTER_PHASE_SWAP_REL_FILES); + pgstat_progress_update_param(PROGRESS_REPACK_PHASE, + PROGRESS_REPACK_PHASE_SWAP_REL_FILES); /* Zero out possible results from swapped_relation_files */ memset(mapped_tables, 0, sizeof(mapped_tables)); @@ -1509,14 +1512,14 @@ finish_heap_swap(Oid OIDOldHeap, Oid OIDNewHeap, reindex_flags |= REINDEX_REL_FORCE_INDEXES_PERMANENT; /* Report that we are now reindexing relations */ - pgstat_progress_update_param(PROGRESS_CLUSTER_PHASE, - PROGRESS_CLUSTER_PHASE_REBUILD_INDEX); + pgstat_progress_update_param(PROGRESS_REPACK_PHASE, + PROGRESS_REPACK_PHASE_REBUILD_INDEX); reindex_relation(NULL, OIDOldHeap, reindex_flags, &reindex_params); /* Report that we are now doing clean up */ - pgstat_progress_update_param(PROGRESS_CLUSTER_PHASE, - PROGRESS_CLUSTER_PHASE_FINAL_CLEANUP); + pgstat_progress_update_param(PROGRESS_REPACK_PHASE, + PROGRESS_REPACK_PHASE_FINAL_CLEANUP); /* * If the relation being rebuilt is pg_class, swap_relation_files() @@ -1632,106 +1635,191 @@ finish_heap_swap(Oid OIDOldHeap, Oid OIDNewHeap, } } - /* - * Get a list of tables that the current user has privileges on and - * have indisclustered set. Return the list in a List * of RelToCluster - * (stored in the specified memory context), each one giving the tableOid - * and the indexOid on which the table is already clustered. + * Determine which relations to process, when REPACK/CLUSTER is called + * without specifying a table name. The exact process depends on whether + * USING INDEX was given or not, and in any case we only return tables and + * materialized views that the current user has privileges to repack/cluster. + * + * If USING INDEX was given, we scan pg_index to find those that have + * indisclustered set; if it was not given, scan pg_class and return all + * tables. + * + * Return it as a list of RelToCluster in the given memory context. */ static List * -get_tables_to_cluster(MemoryContext cluster_context) +get_tables_to_repack(RepackCommand cmd, bool usingindex, MemoryContext permcxt) { - Relation indRelation; + Relation catalog; TableScanDesc scan; - ScanKeyData entry; - HeapTuple indexTuple; - Form_pg_index index; - MemoryContext old_context; + HeapTuple tuple; List *rtcs = NIL; - /* - * Get all indexes that have indisclustered set and that the current user - * has the appropriate privileges for. - */ - indRelation = table_open(IndexRelationId, AccessShareLock); - ScanKeyInit(&entry, - Anum_pg_index_indisclustered, - BTEqualStrategyNumber, F_BOOLEQ, - BoolGetDatum(true)); - scan = table_beginscan_catalog(indRelation, 1, &entry); - while ((indexTuple = heap_getnext(scan, ForwardScanDirection)) != NULL) + if (usingindex) { - RelToCluster *rtc; + ScanKeyData entry; - index = (Form_pg_index) GETSTRUCT(indexTuple); + catalog = table_open(IndexRelationId, AccessShareLock); + ScanKeyInit(&entry, + Anum_pg_index_indisclustered, + BTEqualStrategyNumber, F_BOOLEQ, + BoolGetDatum(true)); + scan = table_beginscan_catalog(catalog, 1, &entry); + while ((tuple = heap_getnext(scan, ForwardScanDirection)) != NULL) + { + RelToCluster *rtc; + Form_pg_index index; + MemoryContext oldcxt; - if (!cluster_is_permitted_for_relation(index->indrelid, GetUserId())) - continue; + index = (Form_pg_index) GETSTRUCT(tuple); - /* Use a permanent memory context for the result list */ - old_context = MemoryContextSwitchTo(cluster_context); + /* + * Try to obtain a light lock on the index's table, to ensure it + * doesn't go away while we collect the list. If we cannot, just + * disregard it. + */ + if (!ConditionalLockRelationOid(index->indrelid, AccessShareLock)) + continue; - rtc = palloc_object(RelToCluster); - rtc->tableOid = index->indrelid; - rtc->indexOid = index->indexrelid; - rtcs = lappend(rtcs, rtc); + /* Verify that the table still exists */ + if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(index->indrelid))) + { + /* Release useless lock */ + UnlockRelationOid(index->indrelid, AccessShareLock); + continue; + } - MemoryContextSwitchTo(old_context); + if (!cluster_is_permitted_for_relation(cmd, index->indrelid, + GetUserId())) + continue; + + /* Use a permanent memory context for the result list */ + oldcxt = MemoryContextSwitchTo(permcxt); + rtc = palloc_object(RelToCluster); + rtc->tableOid = index->indrelid; + rtc->indexOid = index->indexrelid; + rtcs = lappend(rtcs, rtc); + MemoryContextSwitchTo(oldcxt); + } } - table_endscan(scan); + else + { + catalog = table_open(RelationRelationId, AccessShareLock); + scan = table_beginscan_catalog(catalog, 0, NULL); - relation_close(indRelation, AccessShareLock); + while ((tuple = heap_getnext(scan, ForwardScanDirection)) != NULL) + { + RelToCluster *rtc; + Form_pg_class class; + MemoryContext oldcxt; + + class = (Form_pg_class) GETSTRUCT(tuple); + + /* + * Try to obtain a light lock on the table, to ensure it doesn't + * go away while we collect the list. If we cannot, just + * disregard the table. + */ + if (!ConditionalLockRelationOid(class->oid, AccessShareLock)) + continue; + + /* Verify that the table still exists */ + if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(class->oid))) + { + /* Release useless lock */ + UnlockRelationOid(class->oid, AccessShareLock); + continue; + } + + /* Can only process plain tables and matviews */ + if (class->relkind != RELKIND_RELATION && + class->relkind != RELKIND_MATVIEW) + continue; + + /* noisily skip rels which the user can't process */ + if (!cluster_is_permitted_for_relation(cmd, class->oid, + GetUserId())) + continue; + + /* Use a permanent memory context for the result list */ + oldcxt = MemoryContextSwitchTo(permcxt); + rtc = palloc_object(RelToCluster); + rtc->tableOid = class->oid; + rtc->indexOid = InvalidOid; + rtcs = lappend(rtcs, rtc); + MemoryContextSwitchTo(oldcxt); + } + } + + table_endscan(scan); + relation_close(catalog, AccessShareLock); return rtcs; } /* - * Given an index on a partitioned table, return a list of RelToCluster for + * Given a partitioned table or its index, return a list of RelToCluster for * all the children leaves tables/indexes. * * Like expand_vacuum_rel, but here caller must hold AccessExclusiveLock * on the table containing the index. + * + * 'rel_is_index' tells whether 'relid' is that of an index (true) or of the + * owning relation. */ static List * -get_tables_to_cluster_partitioned(MemoryContext cluster_context, Oid indexOid) +get_tables_to_repack_partitioned(RepackCommand cmd, Oid relid, + bool rel_is_index, MemoryContext permcxt) { List *inhoids; - ListCell *lc; List *rtcs = NIL; - MemoryContext old_context; - /* Do not lock the children until they're processed */ - inhoids = find_all_inheritors(indexOid, NoLock, NULL); - - foreach(lc, inhoids) + /* + * Do not lock the children until they're processed. Note that we do hold + * a lock on the parent partitioned table. + */ + inhoids = find_all_inheritors(relid, NoLock, NULL); + foreach_oid(child_oid, inhoids) { - Oid indexrelid = lfirst_oid(lc); - Oid relid = IndexGetRelation(indexrelid, false); + Oid table_oid, + index_oid; RelToCluster *rtc; + MemoryContext oldcxt; - /* consider only leaf indexes */ - if (get_rel_relkind(indexrelid) != RELKIND_INDEX) - continue; + if (rel_is_index) + { + /* consider only leaf indexes */ + if (get_rel_relkind(child_oid) != RELKIND_INDEX) + continue; + + table_oid = IndexGetRelation(child_oid, false); + index_oid = child_oid; + } + else + { + /* consider only leaf relations */ + if (get_rel_relkind(child_oid) != RELKIND_RELATION) + continue; + + table_oid = child_oid; + index_oid = InvalidOid; + } /* * It's possible that the user does not have privileges to CLUSTER the - * leaf partition despite having such privileges on the partitioned - * table. We skip any partitions which the user is not permitted to - * CLUSTER. + * leaf partition despite having them on the partitioned table. Skip + * if so. */ - if (!cluster_is_permitted_for_relation(relid, GetUserId())) + if (!cluster_is_permitted_for_relation(cmd, table_oid, GetUserId())) continue; /* Use a permanent memory context for the result list */ - old_context = MemoryContextSwitchTo(cluster_context); - + oldcxt = MemoryContextSwitchTo(permcxt); rtc = palloc_object(RelToCluster); - rtc->tableOid = relid; - rtc->indexOid = indexrelid; + rtc->tableOid = table_oid; + rtc->indexOid = index_oid; rtcs = lappend(rtcs, rtc); - - MemoryContextSwitchTo(old_context); + MemoryContextSwitchTo(oldcxt); } return rtcs; @@ -1742,13 +1830,167 @@ get_tables_to_cluster_partitioned(MemoryContext cluster_context, Oid indexOid) * function emits a WARNING. */ static bool -cluster_is_permitted_for_relation(Oid relid, Oid userid) +cluster_is_permitted_for_relation(RepackCommand cmd, Oid relid, Oid userid) { + Assert(cmd == REPACK_COMMAND_CLUSTER || cmd == REPACK_COMMAND_REPACK); + if (pg_class_aclcheck(relid, userid, ACL_MAINTAIN) == ACLCHECK_OK) return true; ereport(WARNING, - (errmsg("permission denied to cluster \"%s\", skipping it", - get_rel_name(relid)))); + errmsg("permission denied to execute %s on \"%s\", skipping it", + RepackCommandAsString(cmd), + get_rel_name(relid))); + return false; } + + +/* + * Given a RepackStmt with an indicated relation name, resolve the relation + * name, obtain lock on it, then determine what to do based on the relation + * type: if it's table and not partitioned, repack it as indicated (using an + * existing clustered index, or following the given one), and return NULL. + * + * On the other hand, if the table is partitioned, do nothing further and + * instead return the opened and locked relcache entry, so that caller can + * process the partitions using the multiple-table handling code. In this + * case, if an index name is given, it's up to the caller to resolve it. + */ +static Relation +process_single_relation(RepackStmt *stmt, ClusterParams *params) +{ + Relation rel; + Oid tableOid; + + Assert(stmt->relation != NULL); + Assert(stmt->command == REPACK_COMMAND_CLUSTER || + stmt->command == REPACK_COMMAND_REPACK); + + /* + * Find, lock, and check permissions on the table. We obtain + * AccessExclusiveLock right away to avoid lock-upgrade hazard in the + * single-transaction case. + */ + tableOid = RangeVarGetRelidExtended(stmt->relation->relation, + AccessExclusiveLock, + 0, + RangeVarCallbackMaintainsTable, + NULL); + rel = table_open(tableOid, NoLock); + + /* + * Reject clustering a remote temp table ... their local buffer manager is + * not going to cope. + */ + if (RELATION_IS_OTHER_TEMP(rel)) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot execute %s on temporary tables of other sessions", + RepackCommandAsString(stmt->command))); + + /* + * Make sure ANALYZE is specified if a column list is present. + */ + if ((params->options & CLUOPT_ANALYZE) == 0 && stmt->relation->va_cols != NIL) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ANALYZE option must be specified when a column list is provided")); + + /* + * For partitioned tables, let caller handle this. Otherwise, process it + * here and we're done. + */ + if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + return rel; + else + { + Oid indexOid; + + indexOid = determine_clustered_index(rel, stmt->usingindex, + stmt->indexname); + if (OidIsValid(indexOid)) + check_index_is_clusterable(rel, indexOid, AccessExclusiveLock); + cluster_rel(stmt->command, rel, indexOid, params); + + /* Do an analyze, if requested */ + if (params->options & CLUOPT_ANALYZE) + { + VacuumParams vac_params = {0}; + + vac_params.options |= VACOPT_ANALYZE; + if (params->options & CLUOPT_VERBOSE) + vac_params.options |= VACOPT_VERBOSE; + analyze_rel(tableOid, NULL, vac_params, + stmt->relation->va_cols, true, NULL); + } + + return NULL; + } +} + +/* + * Given a relation and the usingindex/indexname options in a + * REPACK USING INDEX or CLUSTER command, return the OID of the + * index to use for clustering the table. + * + * Caller must hold lock on the relation so that the set of indexes + * doesn't change, and must call check_index_is_clusterable. + */ +static Oid +determine_clustered_index(Relation rel, bool usingindex, const char *indexname) +{ + Oid indexOid; + + if (indexname == NULL && usingindex) + { + /* + * If USING INDEX with no name is given, find a clustered index, or + * error out if none. + */ + indexOid = InvalidOid; + foreach_oid(idxoid, RelationGetIndexList(rel)) + { + if (get_index_isclustered(idxoid)) + { + indexOid = idxoid; + break; + } + } + + if (!OidIsValid(indexOid)) + ereport(ERROR, + errcode(ERRCODE_UNDEFINED_OBJECT), + errmsg("there is no previously clustered index for table \"%s\"", + RelationGetRelationName(rel))); + } + else if (indexname != NULL) + { + /* An index was specified; obtain its OID. */ + indexOid = get_relname_relid(indexname, rel->rd_rel->relnamespace); + if (!OidIsValid(indexOid)) + ereport(ERROR, + errcode(ERRCODE_UNDEFINED_OBJECT), + errmsg("index \"%s\" for table \"%s\" does not exist", + indexname, RelationGetRelationName(rel))); + } + else + indexOid = InvalidOid; + + return indexOid; +} + +static const char * +RepackCommandAsString(RepackCommand cmd) +{ + switch (cmd) + { + case REPACK_COMMAND_REPACK: + return "REPACK"; + case REPACK_COMMAND_VACUUMFULL: + return "VACUUM"; + case REPACK_COMMAND_CLUSTER: + return "CLUSTER"; + } + return "???"; /* keep compiler quiet */ +} diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c index 03932f45c8a..aea998260e1 100644 --- a/src/backend/commands/vacuum.c +++ b/src/backend/commands/vacuum.c @@ -351,7 +351,6 @@ ExecVacuum(ParseState *pstate, VacuumStmt *vacstmt, bool isTopLevel) } } - /* * Sanity check DISABLE_PAGE_SKIPPING option. */ @@ -2289,8 +2288,9 @@ vacuum_rel(Oid relid, RangeVar *relation, VacuumParams params, if ((params.options & VACOPT_VERBOSE) != 0) cluster_params.options |= CLUOPT_VERBOSE; - /* VACUUM FULL is now a variant of CLUSTER; see cluster.c */ - cluster_rel(rel, InvalidOid, &cluster_params); + /* VACUUM FULL is a variant of REPACK; see cluster.c */ + cluster_rel(REPACK_COMMAND_VACUUMFULL, rel, InvalidOid, + &cluster_params); /* cluster_rel closes the relation, but keeps lock */ rel = NULL; diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index c567252acc4..86f043523ac 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -287,7 +287,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); AlterCompositeTypeStmt AlterUserMappingStmt AlterRoleStmt AlterRoleSetStmt AlterPolicyStmt AlterStatsStmt AlterDefaultPrivilegesStmt DefACLAction - AnalyzeStmt CallStmt ClosePortalStmt ClusterStmt CommentStmt + AnalyzeStmt CallStmt ClosePortalStmt CommentStmt ConstraintsSetStmt CopyStmt CreateAsStmt CreateCastStmt CreateDomainStmt CreateExtensionStmt CreateGroupStmt CreateOpClassStmt CreateOpFamilyStmt AlterOpFamilyStmt CreatePLangStmt @@ -304,7 +304,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); GrantStmt GrantRoleStmt ImportForeignSchemaStmt IndexStmt InsertStmt ListenStmt LoadStmt LockStmt MergeStmt NotifyStmt ExplainableStmt PreparableStmt CreateFunctionStmt AlterFunctionStmt ReindexStmt RemoveAggrStmt - RemoveFuncStmt RemoveOperStmt RenameStmt ReturnStmt RevokeStmt RevokeRoleStmt + RemoveFuncStmt RemoveOperStmt RenameStmt RepackStmt ReturnStmt RevokeStmt RevokeRoleStmt RuleActionStmt RuleActionStmtOrEmpty RuleStmt SecLabelStmt SelectStmt TransactionStmt TransactionStmtLegacy TruncateStmt UnlistenStmt UpdateStmt VacuumStmt @@ -323,7 +323,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <str> opt_single_name %type <list> opt_qualified_name -%type <boolean> opt_concurrently +%type <boolean> opt_concurrently opt_usingindex %type <dbehavior> opt_drop_behavior %type <list> opt_utility_option_list %type <list> opt_wait_with_clause @@ -773,7 +773,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); QUOTE QUOTES RANGE READ REAL REASSIGN RECURSIVE REF_P REFERENCES REFERENCING - REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA + REFRESH REINDEX RELATIVE_P RELEASE RENAME REPACK REPEATABLE REPLACE REPLICA RESET RESPECT_P RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP ROUTINE ROUTINES ROW ROWS RULE @@ -1035,7 +1035,6 @@ stmt: | CallStmt | CheckPointStmt | ClosePortalStmt - | ClusterStmt | CommentStmt | ConstraintsSetStmt | CopyStmt @@ -1109,6 +1108,7 @@ stmt: | RemoveFuncStmt | RemoveOperStmt | RenameStmt + | RepackStmt | RevokeStmt | RevokeRoleStmt | RuleStmt @@ -1146,6 +1146,11 @@ opt_concurrently: | /*EMPTY*/ { $$ = false; } ; +opt_usingindex: + USING INDEX { $$ = true; } + | /* EMPTY */ { $$ = false; } + ; + opt_drop_behavior: CASCADE { $$ = DROP_CASCADE; } | RESTRICT { $$ = DROP_RESTRICT; } @@ -12036,38 +12041,82 @@ CreateConversionStmt: /***************************************************************************** * * QUERY: + * REPACK [ (options) ] [ <qualified_name> [ <name_list> ] [ USING INDEX <index_name> ] ] + * + * obsolete variants: * CLUSTER (options) [ <qualified_name> [ USING <index_name> ] ] * CLUSTER [VERBOSE] [ <qualified_name> [ USING <index_name> ] ] * CLUSTER [VERBOSE] <index_name> ON <qualified_name> (for pre-8.3) * *****************************************************************************/ -ClusterStmt: - CLUSTER '(' utility_option_list ')' qualified_name cluster_index_specification +RepackStmt: + REPACK opt_utility_option_list vacuum_relation USING INDEX name { - ClusterStmt *n = makeNode(ClusterStmt); + RepackStmt *n = makeNode(RepackStmt); - n->relation = $5; + n->command = REPACK_COMMAND_REPACK; + n->relation = (VacuumRelation *) $3; n->indexname = $6; + n->usingindex = true; + n->params = $2; + $$ = (Node *) n; + } + | REPACK opt_utility_option_list vacuum_relation opt_usingindex + { + RepackStmt *n = makeNode(RepackStmt); + + n->command = REPACK_COMMAND_REPACK; + n->relation = (VacuumRelation *) $3; + n->indexname = NULL; + n->usingindex = $4; + n->params = $2; + $$ = (Node *) n; + } + | REPACK opt_utility_option_list opt_usingindex + { + RepackStmt *n = makeNode(RepackStmt); + + n->command = REPACK_COMMAND_REPACK; + n->relation = NULL; + n->indexname = NULL; + n->usingindex = $3; + n->params = $2; + $$ = (Node *) n; + } + | CLUSTER '(' utility_option_list ')' qualified_name cluster_index_specification + { + RepackStmt *n = makeNode(RepackStmt); + + n->command = REPACK_COMMAND_CLUSTER; + n->relation = makeNode(VacuumRelation); + n->relation->relation = $5; + n->indexname = $6; + n->usingindex = true; n->params = $3; $$ = (Node *) n; } | CLUSTER opt_utility_option_list { - ClusterStmt *n = makeNode(ClusterStmt); + RepackStmt *n = makeNode(RepackStmt); + n->command = REPACK_COMMAND_CLUSTER; n->relation = NULL; n->indexname = NULL; + n->usingindex = true; n->params = $2; $$ = (Node *) n; } /* unparenthesized VERBOSE kept for pre-14 compatibility */ | CLUSTER opt_verbose qualified_name cluster_index_specification { - ClusterStmt *n = makeNode(ClusterStmt); + RepackStmt *n = makeNode(RepackStmt); - n->relation = $3; + n->command = REPACK_COMMAND_CLUSTER; + n->relation = makeNode(VacuumRelation); + n->relation->relation = $3; n->indexname = $4; + n->usingindex = true; if ($2) n->params = list_make1(makeDefElem("verbose", NULL, @2)); $$ = (Node *) n; @@ -12075,20 +12124,25 @@ ClusterStmt: /* unparenthesized VERBOSE kept for pre-17 compatibility */ | CLUSTER VERBOSE { - ClusterStmt *n = makeNode(ClusterStmt); + RepackStmt *n = makeNode(RepackStmt); + n->command = REPACK_COMMAND_CLUSTER; n->relation = NULL; n->indexname = NULL; + n->usingindex = true; n->params = list_make1(makeDefElem("verbose", NULL, @2)); $$ = (Node *) n; } /* kept for pre-8.3 compatibility */ | CLUSTER opt_verbose name ON qualified_name { - ClusterStmt *n = makeNode(ClusterStmt); + RepackStmt *n = makeNode(RepackStmt); - n->relation = $5; + n->command = REPACK_COMMAND_CLUSTER; + n->relation = makeNode(VacuumRelation); + n->relation->relation = $5; n->indexname = $3; + n->usingindex = true; if ($2) n->params = list_make1(makeDefElem("verbose", NULL, @2)); $$ = (Node *) n; @@ -18145,6 +18199,7 @@ unreserved_keyword: | RELATIVE_P | RELEASE | RENAME + | REPACK | REPEATABLE | REPLACE | REPLICA @@ -18782,6 +18837,7 @@ bare_label_keyword: | RELATIVE_P | RELEASE | RENAME + | REPACK | REPEATABLE | REPLACE | REPLICA diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index bf707f2d57f..b0032ab5eee 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -279,9 +279,9 @@ ClassifyUtilityCommandAsReadOnly(Node *parsetree) return COMMAND_OK_IN_RECOVERY | COMMAND_OK_IN_READ_ONLY_TXN; } - case T_ClusterStmt: case T_ReindexStmt: case T_VacuumStmt: + case T_RepackStmt: { /* * These commands write WAL, so they're not strictly @@ -856,14 +856,14 @@ standard_ProcessUtility(PlannedStmt *pstmt, ExecuteCallStmt(castNode(CallStmt, parsetree), params, isAtomicContext, dest); break; - case T_ClusterStmt: - cluster(pstate, (ClusterStmt *) parsetree, isTopLevel); - break; - case T_VacuumStmt: ExecVacuum(pstate, (VacuumStmt *) parsetree, isTopLevel); break; + case T_RepackStmt: + ExecRepack(pstate, (RepackStmt *) parsetree, isTopLevel); + break; + case T_ExplainStmt: ExplainQuery(pstate, (ExplainStmt *) parsetree, params, dest); break; @@ -2865,10 +2865,6 @@ CreateCommandTag(Node *parsetree) tag = CMDTAG_CALL; break; - case T_ClusterStmt: - tag = CMDTAG_CLUSTER; - break; - case T_VacuumStmt: if (((VacuumStmt *) parsetree)->is_vacuumcmd) tag = CMDTAG_VACUUM; @@ -2876,6 +2872,13 @@ CreateCommandTag(Node *parsetree) tag = CMDTAG_ANALYZE; break; + case T_RepackStmt: + if (((RepackStmt *) parsetree)->command == REPACK_COMMAND_CLUSTER) + tag = CMDTAG_CLUSTER; + else + tag = CMDTAG_REPACK; + break; + case T_ExplainStmt: tag = CMDTAG_EXPLAIN; break; @@ -3517,7 +3520,7 @@ GetCommandLogLevel(Node *parsetree) lev = LOGSTMT_ALL; break; - case T_ClusterStmt: + case T_RepackStmt: lev = LOGSTMT_DDL; break; diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c index b1df96e7b0b..15accbce2ba 100644 --- a/src/backend/utils/adt/pgstatfuncs.c +++ b/src/backend/utils/adt/pgstatfuncs.c @@ -287,8 +287,8 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS) cmdtype = PROGRESS_COMMAND_VACUUM; else if (pg_strcasecmp(cmd, "ANALYZE") == 0) cmdtype = PROGRESS_COMMAND_ANALYZE; - else if (pg_strcasecmp(cmd, "CLUSTER") == 0) - cmdtype = PROGRESS_COMMAND_CLUSTER; + else if (pg_strcasecmp(cmd, "REPACK") == 0) + cmdtype = PROGRESS_COMMAND_REPACK; else if (pg_strcasecmp(cmd, "CREATE INDEX") == 0) cmdtype = PROGRESS_COMMAND_CREATE_INDEX; else if (pg_strcasecmp(cmd, "BASEBACKUP") == 0) diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index 8b91bc00062..2a1bb47ff03 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -1267,7 +1267,7 @@ static const char *const sql_commands[] = { "DELETE FROM", "DISCARD", "DO", "DROP", "END", "EXECUTE", "EXPLAIN", "FETCH", "GRANT", "IMPORT FOREIGN SCHEMA", "INSERT INTO", "LISTEN", "LOAD", "LOCK", "MERGE INTO", "MOVE", "NOTIFY", "PREPARE", - "REASSIGN", "REFRESH MATERIALIZED VIEW", "REINDEX", "RELEASE", + "REASSIGN", "REFRESH MATERIALIZED VIEW", "REINDEX", "RELEASE", "REPACK", "RESET", "REVOKE", "ROLLBACK", "SAVEPOINT", "SECURITY LABEL", "SELECT", "SET", "SHOW", "START", "TABLE", "TRUNCATE", "UNLISTEN", "UPDATE", "VACUUM", "VALUES", @@ -5086,6 +5086,46 @@ match_previous_words(int pattern_id, COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces); } +/* REPACK */ + else if (Matches("REPACK")) + COMPLETE_WITH_SCHEMA_QUERY_PLUS(Query_for_list_of_clusterables, + "(", "USING INDEX"); + else if (Matches("REPACK", "(*)")) + COMPLETE_WITH_SCHEMA_QUERY_PLUS(Query_for_list_of_clusterables, + "USING INDEX"); + else if (Matches("REPACK", MatchAnyExcept("("))) + COMPLETE_WITH("USING INDEX"); + else if (Matches("REPACK", "(*)", MatchAnyExcept("("))) + COMPLETE_WITH("USING INDEX"); + else if (Matches("REPACK", MatchAny, "USING", "INDEX") || + Matches("REPACK", "(*)", MatchAny, "USING", "INDEX")) + { + set_completion_reference(prev3_wd); + COMPLETE_WITH_SCHEMA_QUERY(Query_for_index_of_table); + } + /* + * Complete ... [ (*) ] <sth> USING INDEX, with a list of indexes for + * <sth>. + */ + else if (TailMatches(MatchAny, "USING", "INDEX")) + { + set_completion_reference(prev3_wd); + COMPLETE_WITH_SCHEMA_QUERY(Query_for_index_of_table); + } + else if (HeadMatches("REPACK", "(*") && + !HeadMatches("REPACK", "(*)")) + { + /* + * This fires if we're in an unfinished parenthesized option list. + * get_previous_words treats a completed parenthesized option list as + * one word, so the above test is correct. + */ + if (ends_with(prev_wd, '(') || ends_with(prev_wd, ',')) + COMPLETE_WITH("ANALYZE", "VERBOSE"); + else if (TailMatches("ANALYZE", "VERBOSE")) + COMPLETE_WITH("ON", "OFF"); + } + /* SECURITY LABEL */ else if (Matches("SECURITY")) COMPLETE_WITH("LABEL"); diff --git a/src/include/commands/cluster.h b/src/include/commands/cluster.h index 8ea81622f9d..28741988478 100644 --- a/src/include/commands/cluster.h +++ b/src/include/commands/cluster.h @@ -24,6 +24,7 @@ #define CLUOPT_RECHECK 0x02 /* recheck relation state */ #define CLUOPT_RECHECK_ISCLUSTERED 0x04 /* recheck relation state for * indisclustered */ +#define CLUOPT_ANALYZE 0x08 /* do an ANALYZE */ /* options for CLUSTER */ typedef struct ClusterParams @@ -31,8 +32,11 @@ typedef struct ClusterParams bits32 options; /* bitmask of CLUOPT_* */ } ClusterParams; -extern void cluster(ParseState *pstate, ClusterStmt *stmt, bool isTopLevel); -extern void cluster_rel(Relation OldHeap, Oid indexOid, ClusterParams *params); + +extern void ExecRepack(ParseState *pstate, RepackStmt *stmt, bool isTopLevel); + +extern void cluster_rel(RepackCommand command, Relation OldHeap, Oid indexOid, + ClusterParams *params); extern void check_index_is_clusterable(Relation OldHeap, Oid indexOid, LOCKMODE lockmode); extern void mark_index_clustered(Relation rel, Oid indexOid, bool is_internal); diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h index 359221dc296..f00e39b937d 100644 --- a/src/include/commands/progress.h +++ b/src/include/commands/progress.h @@ -73,28 +73,34 @@ #define PROGRESS_ANALYZE_STARTED_BY_MANUAL 1 #define PROGRESS_ANALYZE_STARTED_BY_AUTOVACUUM 2 -/* Progress parameters for cluster */ -#define PROGRESS_CLUSTER_COMMAND 0 -#define PROGRESS_CLUSTER_PHASE 1 -#define PROGRESS_CLUSTER_INDEX_RELID 2 -#define PROGRESS_CLUSTER_HEAP_TUPLES_SCANNED 3 -#define PROGRESS_CLUSTER_HEAP_TUPLES_WRITTEN 4 -#define PROGRESS_CLUSTER_TOTAL_HEAP_BLKS 5 -#define PROGRESS_CLUSTER_HEAP_BLKS_SCANNED 6 -#define PROGRESS_CLUSTER_INDEX_REBUILD_COUNT 7 +/* + * Progress parameters for REPACK. + * + * Values for PROGRESS_REPACK_COMMAND are defined as in RepackCommand. + * + * Note: Since REPACK shares code with CLUSTER, these values are also + * used by CLUSTER. (CLUSTER is now deprecated, so it makes little sense to + * introduce a separate set of constants.) + */ +#define PROGRESS_REPACK_COMMAND 0 +#define PROGRESS_REPACK_PHASE 1 +#define PROGRESS_REPACK_INDEX_RELID 2 +#define PROGRESS_REPACK_HEAP_TUPLES_SCANNED 3 +#define PROGRESS_REPACK_HEAP_TUPLES_WRITTEN 4 +#define PROGRESS_REPACK_TOTAL_HEAP_BLKS 5 +#define PROGRESS_REPACK_HEAP_BLKS_SCANNED 6 +#define PROGRESS_REPACK_INDEX_REBUILD_COUNT 7 -/* Phases of cluster (as advertised via PROGRESS_CLUSTER_PHASE) */ -#define PROGRESS_CLUSTER_PHASE_SEQ_SCAN_HEAP 1 -#define PROGRESS_CLUSTER_PHASE_INDEX_SCAN_HEAP 2 -#define PROGRESS_CLUSTER_PHASE_SORT_TUPLES 3 -#define PROGRESS_CLUSTER_PHASE_WRITE_NEW_HEAP 4 -#define PROGRESS_CLUSTER_PHASE_SWAP_REL_FILES 5 -#define PROGRESS_CLUSTER_PHASE_REBUILD_INDEX 6 -#define PROGRESS_CLUSTER_PHASE_FINAL_CLEANUP 7 - -/* Commands of PROGRESS_CLUSTER */ -#define PROGRESS_CLUSTER_COMMAND_CLUSTER 1 -#define PROGRESS_CLUSTER_COMMAND_VACUUM_FULL 2 +/* + * Phases of repack (as advertised via PROGRESS_REPACK_PHASE). + */ +#define PROGRESS_REPACK_PHASE_SEQ_SCAN_HEAP 1 +#define PROGRESS_REPACK_PHASE_INDEX_SCAN_HEAP 2 +#define PROGRESS_REPACK_PHASE_SORT_TUPLES 3 +#define PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP 4 +#define PROGRESS_REPACK_PHASE_SWAP_REL_FILES 5 +#define PROGRESS_REPACK_PHASE_REBUILD_INDEX 6 +#define PROGRESS_REPACK_PHASE_FINAL_CLEANUP 7 /* Progress parameters for CREATE INDEX */ /* 3, 4 and 5 reserved for "waitfor" metrics */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 0aec49bdd22..dac839cdf1e 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -3982,18 +3982,6 @@ typedef struct AlterSystemStmt VariableSetStmt *setstmt; /* SET subcommand */ } AlterSystemStmt; -/* ---------------------- - * Cluster Statement (support pbrown's cluster index implementation) - * ---------------------- - */ -typedef struct ClusterStmt -{ - NodeTag type; - RangeVar *relation; /* relation being indexed, or NULL if all */ - char *indexname; /* original index defined */ - List *params; /* list of DefElem nodes */ -} ClusterStmt; - /* ---------------------- * Vacuum and Analyze Statements * @@ -4006,7 +3994,7 @@ typedef struct VacuumStmt NodeTag type; List *options; /* list of DefElem nodes */ List *rels; /* list of VacuumRelation, or NIL for all */ - bool is_vacuumcmd; /* true for VACUUM, false for ANALYZE */ + bool is_vacuumcmd; /* true for VACUUM, false otherwise */ } VacuumStmt; /* @@ -4024,6 +4012,27 @@ typedef struct VacuumRelation List *va_cols; /* list of column names, or NIL for all */ } VacuumRelation; +/* ---------------------- + * Repack Statement + * ---------------------- + */ +typedef enum RepackCommand +{ + REPACK_COMMAND_CLUSTER = 1, + REPACK_COMMAND_REPACK, + REPACK_COMMAND_VACUUMFULL, +} RepackCommand; + +typedef struct RepackStmt +{ + NodeTag type; + RepackCommand command; /* type of command being run */ + VacuumRelation *relation; /* relation being repacked */ + char *indexname; /* order tuples by this index */ + bool usingindex; /* whether USING INDEX is specified */ + List *params; /* list of DefElem nodes */ +} RepackStmt; + /* ---------------------- * Explain Statement * diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index f7753c5c8a8..6f74a8c05c7 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -377,6 +377,7 @@ PG_KEYWORD("reindex", REINDEX, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("relative", RELATIVE_P, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("release", RELEASE, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("rename", RENAME, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("repack", REPACK, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/tcop/cmdtaglist.h b/src/include/tcop/cmdtaglist.h index 1290c9bab68..652dc61b834 100644 --- a/src/include/tcop/cmdtaglist.h +++ b/src/include/tcop/cmdtaglist.h @@ -196,6 +196,7 @@ PG_CMDTAG(CMDTAG_REASSIGN_OWNED, "REASSIGN OWNED", false, false, false) PG_CMDTAG(CMDTAG_REFRESH_MATERIALIZED_VIEW, "REFRESH MATERIALIZED VIEW", true, false, false) PG_CMDTAG(CMDTAG_REINDEX, "REINDEX", true, false, false) PG_CMDTAG(CMDTAG_RELEASE, "RELEASE", false, false, false) +PG_CMDTAG(CMDTAG_REPACK, "REPACK", false, false, false) PG_CMDTAG(CMDTAG_RESET, "RESET", false, false, false) PG_CMDTAG(CMDTAG_REVOKE, "REVOKE", true, false, false) PG_CMDTAG(CMDTAG_REVOKE_ROLE, "REVOKE ROLE", false, false, false) diff --git a/src/include/utils/backend_progress.h b/src/include/utils/backend_progress.h index 19f63b41431..6300dbd15d5 100644 --- a/src/include/utils/backend_progress.h +++ b/src/include/utils/backend_progress.h @@ -24,10 +24,10 @@ typedef enum ProgressCommandType PROGRESS_COMMAND_INVALID, PROGRESS_COMMAND_VACUUM, PROGRESS_COMMAND_ANALYZE, - PROGRESS_COMMAND_CLUSTER, PROGRESS_COMMAND_CREATE_INDEX, PROGRESS_COMMAND_BASEBACKUP, PROGRESS_COMMAND_COPY, + PROGRESS_COMMAND_REPACK, } ProgressCommandType; #define PGSTAT_NUM_PROGRESS_PARAM 20 diff --git a/src/test/regress/expected/cluster.out b/src/test/regress/expected/cluster.out index 4d40a6809ab..277854418fa 100644 --- a/src/test/regress/expected/cluster.out +++ b/src/test/regress/expected/cluster.out @@ -495,6 +495,43 @@ ALTER TABLE clstrpart SET WITHOUT CLUSTER; ERROR: cannot mark index clustered in partitioned table ALTER TABLE clstrpart CLUSTER ON clstrpart_idx; ERROR: cannot mark index clustered in partitioned table +-- Check that REPACK sets new relfilenodes: it should process exactly the same +-- tables as CLUSTER did. +DROP TABLE old_cluster_info; +DROP TABLE new_cluster_info; +CREATE TEMP TABLE old_cluster_info AS SELECT relname, level, relfilenode, relkind FROM pg_partition_tree('clstrpart'::regclass) AS tree JOIN pg_class c ON c.oid=tree.relid ; +REPACK clstrpart USING INDEX clstrpart_idx; +CREATE TEMP TABLE new_cluster_info AS SELECT relname, level, relfilenode, relkind FROM pg_partition_tree('clstrpart'::regclass) AS tree JOIN pg_class c ON c.oid=tree.relid ; +SELECT relname, old.level, old.relkind, old.relfilenode = new.relfilenode FROM old_cluster_info AS old JOIN new_cluster_info AS new USING (relname) ORDER BY relname COLLATE "C"; + relname | level | relkind | ?column? +-------------+-------+---------+---------- + clstrpart | 0 | p | t + clstrpart1 | 1 | p | t + clstrpart11 | 2 | r | f + clstrpart12 | 2 | p | t + clstrpart2 | 1 | r | f + clstrpart3 | 1 | p | t + clstrpart33 | 2 | r | f +(7 rows) + +-- And finally the same for REPACK w/o index. +DROP TABLE old_cluster_info; +DROP TABLE new_cluster_info; +CREATE TEMP TABLE old_cluster_info AS SELECT relname, level, relfilenode, relkind FROM pg_partition_tree('clstrpart'::regclass) AS tree JOIN pg_class c ON c.oid=tree.relid ; +REPACK clstrpart; +CREATE TEMP TABLE new_cluster_info AS SELECT relname, level, relfilenode, relkind FROM pg_partition_tree('clstrpart'::regclass) AS tree JOIN pg_class c ON c.oid=tree.relid ; +SELECT relname, old.level, old.relkind, old.relfilenode = new.relfilenode FROM old_cluster_info AS old JOIN new_cluster_info AS new USING (relname) ORDER BY relname COLLATE "C"; + relname | level | relkind | ?column? +-------------+-------+---------+---------- + clstrpart | 0 | p | t + clstrpart1 | 1 | p | t + clstrpart11 | 2 | r | f + clstrpart12 | 2 | p | t + clstrpart2 | 1 | r | f + clstrpart3 | 1 | p | t + clstrpart33 | 2 | r | f +(7 rows) + DROP TABLE clstrpart; -- Ownership of partitions is checked CREATE TABLE ptnowner(i int unique) PARTITION BY LIST (i); @@ -513,7 +550,7 @@ CREATE TEMP TABLE ptnowner_oldnodes AS JOIN pg_class AS c ON c.oid=tree.relid; SET SESSION AUTHORIZATION regress_ptnowner; CLUSTER ptnowner USING ptnowner_i_idx; -WARNING: permission denied to cluster "ptnowner2", skipping it +WARNING: permission denied to execute CLUSTER on "ptnowner2", skipping it RESET SESSION AUTHORIZATION; SELECT a.relname, a.relfilenode=b.relfilenode FROM pg_class a JOIN ptnowner_oldnodes b USING (oid) ORDER BY a.relname COLLATE "C"; @@ -665,6 +702,101 @@ SELECT * FROM clstr_expression WHERE -a = -3 ORDER BY -a, b; (4 rows) COMMIT; +---------------------------------------------------------------------- +-- +-- REPACK +-- +---------------------------------------------------------------------- +-- REPACK handles individual tables identically to CLUSTER, but it's worth +-- checking if it handles table hierarchies identically as well. +REPACK clstr_tst USING INDEX clstr_tst_c; +-- Verify that inheritance link still works +INSERT INTO clstr_tst_inh VALUES (0, 100, 'in child table 2'); +SELECT a,b,c,substring(d for 30), length(d) from clstr_tst; + a | b | c | substring | length +----+-----+------------------+--------------------------------+-------- + 10 | 14 | catorce | | + 18 | 5 | cinco | | + 9 | 4 | cuatro | | + 26 | 19 | diecinueve | | + 12 | 18 | dieciocho | | + 30 | 16 | dieciseis | | + 24 | 17 | diecisiete | | + 2 | 10 | diez | | + 23 | 12 | doce | | + 11 | 2 | dos | | + 25 | 9 | nueve | | + 31 | 8 | ocho | | + 1 | 11 | once | | + 28 | 15 | quince | | + 32 | 6 | seis | xyzzyxyzzyxyzzyxyzzyxyzzyxyzzy | 500000 + 29 | 7 | siete | | + 15 | 13 | trece | | + 22 | 30 | treinta | | + 17 | 32 | treinta y dos | | + 3 | 31 | treinta y uno | | + 5 | 3 | tres | | + 20 | 1 | uno | | + 6 | 20 | veinte | | + 14 | 25 | veinticinco | | + 21 | 24 | veinticuatro | | + 4 | 22 | veintidos | | + 19 | 29 | veintinueve | | + 16 | 28 | veintiocho | | + 27 | 26 | veintiseis | | + 13 | 27 | veintisiete | | + 7 | 23 | veintitres | | + 8 | 21 | veintiuno | | + 0 | 100 | in child table | | + 0 | 100 | in child table 2 | | +(34 rows) + +-- Verify that foreign key link still works +INSERT INTO clstr_tst (b, c) VALUES (1111, 'this should fail'); +ERROR: insert or update on table "clstr_tst" violates foreign key constraint "clstr_tst_con" +DETAIL: Key (b)=(1111) is not present in table "clstr_tst_s". +SELECT conname FROM pg_constraint WHERE conrelid = 'clstr_tst'::regclass +ORDER BY 1; + conname +---------------------- + clstr_tst_a_not_null + clstr_tst_con + clstr_tst_pkey +(3 rows) + +-- Verify partial analyze works +REPACK (ANALYZE) clstr_tst (a); +REPACK (ANALYZE) clstr_tst; +REPACK (VERBOSE) clstr_tst (a); +ERROR: ANALYZE option must be specified when a column list is provided +-- REPACK w/o argument performs no ordering, so we can only check which tables +-- have the relfilenode changed. +RESET SESSION AUTHORIZATION; +CREATE TEMP TABLE relnodes_old AS +(SELECT relname, relfilenode +FROM pg_class +WHERE relname IN ('clstr_1', 'clstr_2', 'clstr_3')); +SET SESSION AUTHORIZATION regress_clstr_user; +SET client_min_messages = ERROR; -- order of "skipping" warnings may vary +REPACK; +RESET client_min_messages; +RESET SESSION AUTHORIZATION; +CREATE TEMP TABLE relnodes_new AS +(SELECT relname, relfilenode +FROM pg_class +WHERE relname IN ('clstr_1', 'clstr_2', 'clstr_3')); +-- Do the actual comparison. Unlike CLUSTER, clstr_3 should have been +-- processed because there is nothing like clustering index here. +SELECT o.relname FROM relnodes_old o +JOIN relnodes_new n ON o.relname = n.relname +WHERE o.relfilenode <> n.relfilenode +ORDER BY o.relname; + relname +--------- + clstr_1 + clstr_3 +(2 rows) + -- clean up DROP TABLE clustertest; DROP TABLE clstr_1; diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 78a37d9fc8f..6827166afac 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -2002,34 +2002,23 @@ pg_stat_progress_basebackup| SELECT pid, ELSE NULL::text END AS backup_type FROM pg_stat_get_progress_info('BASEBACKUP'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20); -pg_stat_progress_cluster| SELECT s.pid, - s.datid, - d.datname, - s.relid, - CASE s.param1 - WHEN 1 THEN 'CLUSTER'::text - WHEN 2 THEN 'VACUUM FULL'::text - ELSE NULL::text +pg_stat_progress_cluster| SELECT pid, + datid, + datname, + relid, + CASE + WHEN (command = ANY (ARRAY['CLUSTER'::text, 'VACUUM FULL'::text])) THEN command + WHEN (repack_index_relid = (0)::oid) THEN 'VACUUM FULL'::text + ELSE 'CLUSTER'::text END AS command, - CASE s.param2 - WHEN 0 THEN 'initializing'::text - WHEN 1 THEN 'seq scanning heap'::text - WHEN 2 THEN 'index scanning heap'::text - WHEN 3 THEN 'sorting tuples'::text - WHEN 4 THEN 'writing new heap'::text - WHEN 5 THEN 'swapping relation files'::text - WHEN 6 THEN 'rebuilding index'::text - WHEN 7 THEN 'performing final cleanup'::text - ELSE NULL::text - END AS phase, - (s.param3)::oid AS cluster_index_relid, - s.param4 AS heap_tuples_scanned, - s.param5 AS heap_tuples_written, - s.param6 AS heap_blks_total, - s.param7 AS heap_blks_scanned, - s.param8 AS index_rebuild_count - FROM (pg_stat_get_progress_info('CLUSTER'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20) - LEFT JOIN pg_database d ON ((s.datid = d.oid))); + phase, + repack_index_relid AS cluster_index_relid, + heap_tuples_scanned, + heap_tuples_written, + heap_blks_total, + heap_blks_scanned, + index_rebuild_count + FROM pg_stat_progress_repack; pg_stat_progress_copy| SELECT s.pid, s.datid, d.datname, @@ -2089,6 +2078,35 @@ pg_stat_progress_create_index| SELECT s.pid, s.param15 AS partitions_done FROM (pg_stat_get_progress_info('CREATE INDEX'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20) LEFT JOIN pg_database d ON ((s.datid = d.oid))); +pg_stat_progress_repack| SELECT s.pid, + s.datid, + d.datname, + s.relid, + CASE s.param1 + WHEN 1 THEN 'CLUSTER'::text + WHEN 2 THEN 'REPACK'::text + WHEN 3 THEN 'VACUUM FULL'::text + ELSE NULL::text + END AS command, + CASE s.param2 + WHEN 0 THEN 'initializing'::text + WHEN 1 THEN 'seq scanning heap'::text + WHEN 2 THEN 'index scanning heap'::text + WHEN 3 THEN 'sorting tuples'::text + WHEN 4 THEN 'writing new heap'::text + WHEN 5 THEN 'swapping relation files'::text + WHEN 6 THEN 'rebuilding index'::text + WHEN 7 THEN 'performing final cleanup'::text + ELSE NULL::text + END AS phase, + (s.param3)::oid AS repack_index_relid, + s.param4 AS heap_tuples_scanned, + s.param5 AS heap_tuples_written, + s.param6 AS heap_blks_total, + s.param7 AS heap_blks_scanned, + s.param8 AS index_rebuild_count + FROM (pg_stat_get_progress_info('REPACK'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20) + LEFT JOIN pg_database d ON ((s.datid = d.oid))); pg_stat_progress_vacuum| SELECT s.pid, s.datid, d.datname, diff --git a/src/test/regress/sql/cluster.sql b/src/test/regress/sql/cluster.sql index b7115f86104..c976823a3cb 100644 --- a/src/test/regress/sql/cluster.sql +++ b/src/test/regress/sql/cluster.sql @@ -76,7 +76,6 @@ INSERT INTO clstr_tst (b, c) VALUES (1111, 'this should fail'); SELECT conname FROM pg_constraint WHERE conrelid = 'clstr_tst'::regclass ORDER BY 1; - SELECT relname, relkind, EXISTS(SELECT 1 FROM pg_class WHERE oid = c.reltoastrelid) AS hastoast FROM pg_class c WHERE relname LIKE 'clstr_tst%' ORDER BY relname; @@ -229,6 +228,24 @@ SELECT relname, old.level, old.relkind, old.relfilenode = new.relfilenode FROM o CLUSTER clstrpart; ALTER TABLE clstrpart SET WITHOUT CLUSTER; ALTER TABLE clstrpart CLUSTER ON clstrpart_idx; + +-- Check that REPACK sets new relfilenodes: it should process exactly the same +-- tables as CLUSTER did. +DROP TABLE old_cluster_info; +DROP TABLE new_cluster_info; +CREATE TEMP TABLE old_cluster_info AS SELECT relname, level, relfilenode, relkind FROM pg_partition_tree('clstrpart'::regclass) AS tree JOIN pg_class c ON c.oid=tree.relid ; +REPACK clstrpart USING INDEX clstrpart_idx; +CREATE TEMP TABLE new_cluster_info AS SELECT relname, level, relfilenode, relkind FROM pg_partition_tree('clstrpart'::regclass) AS tree JOIN pg_class c ON c.oid=tree.relid ; +SELECT relname, old.level, old.relkind, old.relfilenode = new.relfilenode FROM old_cluster_info AS old JOIN new_cluster_info AS new USING (relname) ORDER BY relname COLLATE "C"; + +-- And finally the same for REPACK w/o index. +DROP TABLE old_cluster_info; +DROP TABLE new_cluster_info; +CREATE TEMP TABLE old_cluster_info AS SELECT relname, level, relfilenode, relkind FROM pg_partition_tree('clstrpart'::regclass) AS tree JOIN pg_class c ON c.oid=tree.relid ; +REPACK clstrpart; +CREATE TEMP TABLE new_cluster_info AS SELECT relname, level, relfilenode, relkind FROM pg_partition_tree('clstrpart'::regclass) AS tree JOIN pg_class c ON c.oid=tree.relid ; +SELECT relname, old.level, old.relkind, old.relfilenode = new.relfilenode FROM old_cluster_info AS old JOIN new_cluster_info AS new USING (relname) ORDER BY relname COLLATE "C"; + DROP TABLE clstrpart; -- Ownership of partitions is checked @@ -313,6 +330,57 @@ EXPLAIN (COSTS OFF) SELECT * FROM clstr_expression WHERE -a = -3 ORDER BY -a, b; SELECT * FROM clstr_expression WHERE -a = -3 ORDER BY -a, b; COMMIT; +---------------------------------------------------------------------- +-- +-- REPACK +-- +---------------------------------------------------------------------- + +-- REPACK handles individual tables identically to CLUSTER, but it's worth +-- checking if it handles table hierarchies identically as well. +REPACK clstr_tst USING INDEX clstr_tst_c; + +-- Verify that inheritance link still works +INSERT INTO clstr_tst_inh VALUES (0, 100, 'in child table 2'); +SELECT a,b,c,substring(d for 30), length(d) from clstr_tst; + +-- Verify that foreign key link still works +INSERT INTO clstr_tst (b, c) VALUES (1111, 'this should fail'); + +SELECT conname FROM pg_constraint WHERE conrelid = 'clstr_tst'::regclass +ORDER BY 1; + +-- Verify partial analyze works +REPACK (ANALYZE) clstr_tst (a); +REPACK (ANALYZE) clstr_tst; +REPACK (VERBOSE) clstr_tst (a); + +-- REPACK w/o argument performs no ordering, so we can only check which tables +-- have the relfilenode changed. +RESET SESSION AUTHORIZATION; +CREATE TEMP TABLE relnodes_old AS +(SELECT relname, relfilenode +FROM pg_class +WHERE relname IN ('clstr_1', 'clstr_2', 'clstr_3')); + +SET SESSION AUTHORIZATION regress_clstr_user; +SET client_min_messages = ERROR; -- order of "skipping" warnings may vary +REPACK; +RESET client_min_messages; + +RESET SESSION AUTHORIZATION; +CREATE TEMP TABLE relnodes_new AS +(SELECT relname, relfilenode +FROM pg_class +WHERE relname IN ('clstr_1', 'clstr_2', 'clstr_3')); + +-- Do the actual comparison. Unlike CLUSTER, clstr_3 should have been +-- processed because there is nothing like clustering index here. +SELECT o.relname FROM relnodes_old o +JOIN relnodes_new n ON o.relname = n.relname +WHERE o.relfilenode <> n.relfilenode +ORDER BY o.relname; + -- clean up DROP TABLE clustertest; DROP TABLE clstr_1; diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 241945734ec..5774e5d4cc9 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2579,6 +2579,8 @@ ReorderBufferTupleCidEnt ReorderBufferTupleCidKey ReorderBufferUpdateProgressTxnCB ReorderTuple +RepackCommand +RepackStmt ReparameterizeForeignPathByChild_function ReplOriginId ReplOriginXactState -- 2.47.3 --kmtyk4ln3767qrma Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v35-0002-Refactor-index_concurrently_create_copy-for-use-.patch" ^ permalink raw reply [nested|flat] 119+ messages in thread
end of thread, other threads:[~2026-02-16 19:49 UTC | newest] Thread overview: 119+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2020-09-08 09:54 [PATCH v21 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH 02/12] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 01/10] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 1/5] Built-in compression method Dilip Kumar <[email protected]> 2021-03-05 04:03 [PATCH 4/8] Built-in compression method Dilip Kumar <[email protected]> 2026-02-16 19:49 [PATCH v35 1/5] Add REPACK command Antonin Houska <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox